Very good idea, but I don't program that way. I use a competent IDE to tell me syntax errors and obvious programming mistakes while I write. If I can't debug my algorithm with trial & error I write a test first and modify my algorithm until it complies with the test.
To think that a programmer will always come up with the exact and most optimal solution to any given problem is a bit naïve. In my 10 years as a programmer I have never seen one wonder-kid that is able to do that. The real wonder kid assets in a team also tend to be those that are more creative than syntax-anal compared to those that can write syntax-error-free-code with pen and pencil and remember large parts of a phonebook...
To think that a programmer will always come up with the exact and most optimal solution to any given problem is a bit naïve
I have no idea where you saw this being implied. The software highlights syntax and lets you check your code as many times as desired, as well as giving hints if your result isn't correct and providing compiler output. Nothing stops you from googling something. It's not like the software will destroy your file and egg your house because your algorithm is o(n2) instead of o(n); that's up to whoever is evaluating the results.
The fact that it has a polish and chinese version should also give you a little hint about the intended audience.
Well, that's one more reason you should like Codility - you can run test compilations (the VERIFY button) which will show you syntax errors. And in bottom left corner you can apply your test data sets. In other words - you work like in your every day environment. Check it out and let me know what you think.
I did, and now I see comments stating that you should be able to run tests on your code while developing it. That's awesome and what I was missing.
Next step intellisense and on-the-fly syntax checking. I hope no programmers click the "compile"-button nowadays to check if their code is correct...
This is quite nice, even if the exercise isn't all that difficult.
It would be nice if the boundary conditions were better specified (people seem to have been bitten by overflowing integers). This could be inferred if you told people that you're using a 32-bit machine (the sum of elements is at most SIZE_MAX * INT_MAX; if int and size_t are 32-bit, SIZE_MAX * INT_MAX < INT64_MAX.)
I was also somewhat surprised to see that my code compiles without #include <stdint.h>. I'm not sure this is correct (it could be solved by putting the code written by the candidate before the test code).
Finally, I assume you have proper sandboxing set up?
I think this is cute but I would think hard about the kinds of questions you develop for this. Coming from the perspective of someone working for a big company with a ton of software people that is not a tech company, I could use something like this in general. But, the question you asked is the kind of thing I ask CS undergrads (almost exactly, in fact) because it allows for clever solutions without actually requiring someone to be a particularly good programmer. I can work through it with them and if the basic solution is easy, prompt them to give me something more clever, so I know how they think, which is good when you're looking at an inexperienced developer. However, for an experienced developer that I want to hire to churn out Java code, the difference to me between a correct but simple O(n^2) solution vs a clever O(n) is not terribly meaningful for determining whether they can actually do what I want them to do. I want something that requires people to write more than a handful of lines and demonstrates an ability to keep track of a few more variables correctly, and honestly I don't care all that much about their algorithmic complexity (although I might care a great deal about their space usage, usage of external resources, etc).
So, if this is the audience you are looking to attract, consider thinking about this as a screening tool for people outside of the startup world that actually need "knows how to really do shit in language X" more than "knows how to write clever algorithm,"
I don't think Codility checks whether one knows how to write clever algorithm. Look at tasks from ACM Collegiate Programming Competitions, they require clever algorithms. Codility deliberately focuses on very fundamental tasks. Also it does not pretend to be a final sign-off, it merely tells you which candidates to reject due to lack of rudimentary programming skills. It's not that surgeons make injections that often in their daily practice, yet I would not trust a surgeon who doesn't know how to make an injection.
I like this comment.
A typical surgeon earns a bachelors degree, then attends medical school, before finally spending three to six years in a residency program (in the United States). This is before becoming a practicing surgeon.
Unfortunately, there is nothing similar (in the United States) that is related to software engineering -- though many here would like to see something similar to an engineering license for technology professionals; there is nothing currently on the horizon (that I know of).
In some ways, I think that Codility can help with this state of affairs. This is particularly interesting when we consider comparing metrics for education and experience; and perhaps 'real-world' success.
The Codility statistical comparison, taking these factors into account, would be very interesting indeed. And could possibly result in data that furthers the state of the art in general.
{In Pascal Language+Assembly}
{$ASMMODE INTEL}
function equi (A : Array of longint; n : longint ) : longint;
var c:Longint;
label noOverflow1;
label noOverflow2;
label ciclo;
label fine;
label Over;
label tot;
Begin
Asm
DEC n
JS over
XOR ECX, ECX {Somma1}
XOR EDI, EDI {Somma2}
XOR EAX, EAX
MOV c, EDI
MOV ESI, n
tot:
MOV EDX, A
MOV EDX, [EDX+ESI*4]
PUSH EDX
ADD ECX, EDX
JNO nooverflow1
ADD c, ECX
nooverflow1:
DEC ESI
JNS tot;
SUB ECX, c
SUB EDI, c
ciclo:
POP EDX
SUB ECX, EDX
CMP ECX, EDI
JE fine
ADD EDI, EDX
JNO nooverflow2
DEC EDI
nooverflow2:
CMP EAX, n
JA over
INC EAX
JMP ciclo
over:
MOV EAX, -1
fine:
end;
End;
Generally we don't want to penalize people too badly for assuming particular (but realistic) type ranges, nevertheless we think that perfect solution should be as portable as possible across different implementations of the language (e.g. one should not assume that C's int is 32 bits wide, since implementation with 16-bit int can still be Standard compliant)
That is an amazing piece of software. I hope you make a mint.
I'd give people access to their language's output facility for playing around to the solution, but that is a minor nitpick. (And I suppose I could open an irb instance on my machine for exploration and then give you code when I'm good and ready.)
My Ruby solution, in case we've got anybody who is struggling: http://pastie.org/771438 This would be my first cut if you asked me to whiteboard this.
The site's very slick. My only feedback is the histogram on the results page could be more prominent - it's cool, useful information but I ended up inspecting the html elements to work out what it was.
Also the results page seemed to keep refreshing for me while waiting for results (Safari 3). Surely that should've been AJAX'd?
+1! Thanks for the reminder. I'm fairly new to python and couldn't remember the keyword. I guess this is a reasonable argument for the time deadline - to get it done quickly I didn't look it up, instead doing the simplest thing I knew would work and thus exposing the pitiful state of my python experience :-)
def equi ( A ):
right = sum(A)
left = 0
for i,x in enumerate(A):
right -= x
if right == left:
return i
left += x
return -1
What's interesting is that all the versions I've seen so far are pretty much identical apart from very minor things. This would seem to me to be obviously the correct way to do it but I'm interested if it's just a way. Did anyone solve this in a different way of equal or greater elegance?
Mine kept one cumulative list adding left to right and another summing right to left. This is conventionally slower, but conceivably faster if you do it in parallel, leaving the final loop to be something like:
for i in xrange(len(A)):
if right[i] == left[i]:
return i
The final loop here is slightly tighter, so as long as your parallelism didn't add big overheads, this will be faster.
I like how the performance evaluation seems to be fairly realistic - I used Ruby and my algorithm could have been more efficient (O(n) space, and O(n) time but with more traversals than necessary), but I scored 100 because I met all the stated constraints. (http://codility.com/demo/results/?id=demo8XKE77-H8G if anyone wants to refer to and/or mock my code.) It seems to me "ability to solve problems well enough under time pressure" is exactly what you want a simple hands-off test like this to screen for.
In face-to-face interviews you'd probably follow up by adding constraints: "your algorithm uses O(n) space in addition to the input, can you do it in constant space?". It would be cool if Codility supported that kind of workflow - following up the submitted solution with refinements on the original question. (Relatedly - do you support the workflow where you get a second try if you didn't get 100, maybe because the evalution revealed a constraint that wasn't clearly stated in the problem outline?)
With paid account it is possible to reopen a task after one solution has been submitted, so the recruiter can "give another chance" to the candidate (although we generally suggest not to), which supports the workflow you describe.
Is the code editor they're using something publicly available? I'm looking around for a light in-browser code editor with programmer-oriented features (e.g. syntax highlighting, this also seems to auto-indent Python code to some degree).
Even I noticed that I had seen something ditto somewhere... Hmm so Codility's editor is copy of the one SPOJ uses, so which one do they both use? Can I too get it :)
If I have an account and am taking a test that I've created, you should provide a link back to the dashboard after I've submitted so I can check how I did. Currently, I'm taken to the the "enter a password" screen and have to manually go back to the homepage and then to the dashboard.
It gave me a test concerning finding the "equilibrium index" of a sequence. It said to assume that the sequence is very long. (The question is repeated here: http://forums.sun.com/thread.jspa?threadID=676596&start=...) This problem seems underspecified, because it doesn't state whether you can iterate repeatedly over the sequence in question. If you can, it's easy. If you can't I'm not sure how to do it.
I assumed that "sequence is very long" meant "the time (and space?) complexity of your solution matters". Iterating over the sequence x times doesn't change the time complexity, so long as x << n (i.e. it's O(n) whether x is 1, 2 or 3).
Unfortunately, revealing whether my assumption proved correct would give away how they assess the solution to this demo problem, which some might view as a spoiler.
EDIT: since people are posting actual solutions, I suppose mere analysis of the testing methodology is okay. My assumption was correct.
Does that ever seem like a good idea? The description suggests it may be a long list. That should give you a hint that you don't need to do this to solve the problem.
The best case possible is to iterate the list once - you sum the list from left to right holding cumulative answers and if the final sum is zero, that index is a valid answer.
The worst case of the best algorithm is to iterate twice - firstly as above, and then again in the opposite direction, comparing values from the first iteration.
Really nice site. Well done. The example problem that showed up to me (equilibrium point detection) was not trivial, but didn't require advanced stuff which a halfway decent programmer wouldn't use day in and day out.
I also like the way you tell me what you scored on and what my result was.
I wonder if you should relax the condition that you can submit only once.
I gather TopCoder offers something similar. What's your strategy to compete with them?
PS: got 94 using C++ in 16 min b/c I didn't check overflow of large numbers. Good thing you check for that.
You know, now that I think about it...The description of the test was pretty incoherent. If I don't know what my test data is, and if the problem description is incoherent, I can't really make it work.
It's also taking a metric fuckton of time to evaluate my solution. Which apparently fails. For reasons I can't debug because I can't see the test data.
Any company that relies on code snippets, incomplete problem statements and undefined test data to judge coders will get what they deserve.
maybe there's more than one task, but the task i took appeared to be completely specified (the one where you find a balance point - i may have remembered the name wrong - for an array, so that the sum of elements to either side is equal).
it wasn't obvious to me what to do on first read, but when i paid a little more attention, everything was there (including edge cases, as far as i could see). in particular, they gave a formal numerical expression that was unambiguous (and which didn't seem to contradict the text).
The spec was clear, but the fact that the test dataset used for the test was the very one given as an example wasn't. plus, it was a bit frustrating not being able to fiddle with the test dataset. I would have done it locally with my compiler, but the necessary boilerplate would have lost me time. So, I didn't test enough, and made a silly error which went undetected because of the trailing zero. I still got 75%, but that looks quite miraculous.
EDIT: It appears I can customize my test dataset, and I didn't see it. Silly me.
Yes, it is even possible to add your own data sets, Codility will tell you what you code returns for them. I guess we should make it more apparent in the interface.
As a student this is very interesting since it could also be used to measure self progress. There's a myriad of ways to test your abilities in most subjects, but if I want to find out if I'm any good at programming or to measure my progress, there are very few easy ways to do that. Are you going to sell this to just business's?
Codility tech team is from Warsaw University, we are more than familiar with TopCoder, ACM CPC, IOI etc. We deliberately keep Codility tasks at fundamental level. Codility is not a competition for 100 world's best programmers, it is a tool to tell decent programmers from poor ones. We can raise the bar arbitrarily high, if there is a customer who needs it.
For C++ it would be nice to know what compiler is being checked against. At least in the demo question I would use a 64 bit int to hold the sum if the arrays are really "large" but MSVC names the type __int64 vs G++ using long long and other such minor details. Same for the other languages aka python 2.x or 3.x etc.
We want to avoid sticking to particular compilers/interpreters. An expert in MSVC should not be penalized only because we use GCC. Also we design the tasks so that they are compiler-agnostic (actually they are even language-agnostic). It is sometimes a bit tricky due to non-standardized type sizes (e.g. C, C++, PHP), but generally the tasks can be solved using very limited and portable subset of a language.
Edit: And won't this be a problem if interviewees can access the internet? The interviewers will have to make sure they can check the browsing history of the computer being used!
I don't mean the interviewers should shut off internet access entirely, but what I mean is that if they do allow access to the internet, they'll need to check the interviewee hasn't just googled for the solution to their codibility test (and they'll also have to make sure the interviewee can't delete their browsing history too).
Codility people, if you're reading, here's a bug report.
The test window's layout is fubar if you use a small window
size. I was using 800x600 and the language selection buttons were hidden behind the source code text box. Something (couldn't tell what) overlapped the first two lines of the instructions.
I'm not sure the timer is a good idea. For one, I found the seconds counting down very distracting. It also encourages quick and inefficient solutions that are easy to hack out, over more efficient algorithms that may take longer to code.
Maybe you should time the programs, instead of the programmers.
We are timing the programs as well - with the scalability assessment. Recruiter can change the timing anyway, but we feel that a bit of pressure is good. You don't always have comfy conditions on the job after all.
I also found the clock annoying. If someone does this for an interview they're likely to feel enough pressure anyways. I'd let the test taker hide it and maybe redisplay it automatically with 5 minutes to go.
As for comfy conditions on the job - I would hope that default conditions are comfy. When interviewing I'm more interested in how people will perform day-to-day rather than in rare situations. Most programming jobs don't really require employees to be "cool under pressure."
Other than that, good job on the execution so far. Reminds me a lot of TopCoder, just applied to a specific niche in a good way. I would even consider using this or recommending it to colleagues for hiring.
Right, but maybe not down to the second. I personally didn't mind, but counting seconds causes a distracting flickering. Maybe you should consider activating it on the last minute only ?
Might need some way of making explicit which libraries are available?
eg. if the programmer goes off and writes a multithreaded solution, they might only realise too late that pthread isn't included. Admittedly they should have checked first using verify, but you never know with interview pressure.
Unless it is stated explicitly, we do not provide anything extra but the vanilla programming language and standard libraries. It is more clear for some languages (e.g. Java, C++) than other (e.g. PHP, JavaScript), but we try to stick to the reasonable canonical set of libraries.
Seems like a cool idea. If they were smart they'd not only target companies looking for people but people looking for jobs. e.g. Adding a "Allow the codility community to take this test" check box. Then you have the a whole world full of people taking your test.
Actually it's there - if you click on "Take a free demo test" (http://codility.com/demo/take-sample-test/) on the Codility page there is a free test and you will be given the option to tweet your result at the end.
Yeah, that demo problem would probably take me at least several hours to solve. I'm bad with sequences and I have a volatile short term memory which means I make mistakes and am slow to process numbers and logic. Interestingly these flaws cause me abhor any over complex logic or code where you have to remember more than 4 or 5 things at once to use properly, which makes my code clean. So maybe I'm not a good programmer (I can't write the complex stuff very easily) but the code I write is usually good. Try testing for that Codility!
Very nice product. Much-needed quantification in tech recruiting. Maybe consider marketing it to the stackoverflow people. They have been running a job board with much fanfare since recently. (94 points for me in 7 mins, yay)
one thing i missed, though, was the ability to write and run my own tests. perhaps a "scratchpad" that evaluates code and returns a result would be a nice addition?
Sorry, some bug on our end - it should be there. We are fixing it right now. In the meantim - if you sign up for a trial account [http://codility.com/accounts/codility-register/] it works fine there (a blue "add a test case" button in the bottom left).
thanks. another smaller point i just remembered - i was a bit surprised to see that the evaluator could see my submissions to the "validate" function. given that you do say they won't influence the score i assumed that they would be discarded.
i can see there are several reasons why this is useful to the evaluator, and i can also see that you're getting close to saturating the user with instructions, but in a perfect world it would be good to have "full disclosure" (i guess you do, in that anyone can run the example....)
