I figuered out, that the "1/log(Played+1.01)" caused the overflow problem.
So the formula without it, is:
10000000000 * (log(Played+1.01) - (log(Skip+1.01))^1.7) / DaysSinceAdded^0.125 / 1.001^( DaysSinceLastPlayed)
I tried to find a formula, which can differ nonplayed-songs with different skips, and which not only give them 0 points.
A possible solution is:
500000000000+10000000000*(log(Played+1.01) / 1.001^( DaysSinceLastPlayed) - (log(Skip+1.01))^1.7) / DaysSinceAdded^0.125
@Berny: I had to move the DaysSinceLastPlayed to the Played, because it isn't defined for never played songs. A redefinition to 0 for 0-played songs in the script would solve that problem.
edit: I just tried with the greatest possible values of plays and skips.
I chose the offset of the formula in such way, that it can't get negative with 9999 skips.
Also i tried to simulate an overflow with very much plays, i gave up at 10^999 (that is a number with 999 zeros, for comparison: we think that our universe contains 7*10^22 stars) and i didn't reach 9999999999999999. my calculator can't handle more
edit2: the customfield has place for points till 10^16-1, but in the script the bound is set to 10^15-1. Is that for purpose or a failure?