PDA

View Full Version : Project CALC - car/track leaderboards for consoles



Maskmagog
13-01-2019, 13:36
Web site updated, see https://projectcalc.org.
Always download latest version of PcalcUpdater here: https://projectcalc.org/download.php
Unzip, run exe, do clean laps in timetrial.
If you enjoy time trial on console, I would love to hear what you think.

You no longer need a Raspberry Pi, the small program that sends your laptimes to the database now runs on a Windows PC too. You can still play on Xbox or PS4 though! :)

The info below is in parts obsolete, from the earlier parts of the project, when everything was run on a local Raspberry Pi. Now most stuff is moved to a server, all you need is the small program linked above.


------------------------------------------------------
TL;DR: Run a Raspberry Pi (very small, cheap computer) that stores your laptimes, fetches PC players laptimes, and displays it all in any browser. Leaderboards per car/track combo for Xbox and PS4!

We all know the sad state of console leaderboards. So with some great help from others, I've built a solution that I call Project CALC – Community Assisted Leaderboards for Consoles. It runs on a Raspberry Pi, and saves all your laps (laptime, sector times etc).

Once the Pi is running, you don't have to do anything. The Pi automatically saves your laps. You can view the leaderboards from any browser. Race Mode changes automatically when you switch car or track. Just leave the Pi running.

This runs on the Pi:

* program that receives UDP data
* database where laps are stored
* Apache2 web server and php pages that displays everything
* Python script that scrapes the cars2-stats-steam page and stores the times in the database

It can use both the old PC1 and the new PC2 udp format.

The program receives and interprets various UDP data from the game. Thank you Zeratall for the library, and all the help! It checks for your current lap record for the car-track combo you are currently running. You could set it to record all laps, or just laps that are valid, or just laps in Time Trial, or whatever you want. Store the data you want. Track temp? Ride height? Tyre temps? Rain density?

The database stores the data, and you could use that data anyway you want. It's a MariaDB, very similar to MySQL, and easy to use.

The php page shows the leaderboards, with dropdown selections for track, car and class. It's also possible to show just your top times for each car for selected track, or show all your stored laps.

263504

263507

There's also a Race Mode, a php page that displays the WR, your best laptime, and the rivals just ahead of you on the leaderboard. All in big LCD letters. This updates automatically if you set a better laptime, or change car or track. I run this on an old tablet next to my monitor, it's really awesome, imho!

263506

This is a huge improvement over in-game leaderboards to me. I'm not a programmer at all, so it has taken a lot of trial and error. I know that there are plenty of computer programs that does this too, probably in a better way. The thing for me is that I don't want my computer turned on all the time, running the program. The Pi on the other hand, can be on all the time. It's slightly larger than a credit card, and costs about $35.

Including data from cars2-stat-steam
I, as many others, have used the cars2-stat-steam page to compare lap times. I found a scraper for this (Thanks Jonas Gulle!), and modified it to scrape all leaderboard pages that I want, and store them in the database. Now i can instantly see where I stand on my tablet: my laptimes are mixed in with the steam laptimes.

For me, the statistics are a big part of racing/gaming. I want to see my lap times and sector times, to analyze it. Now I can! No longer is my (for me) super awesome lap in a Ginetta G40 Junior erased by a crappy lap in a Toyota GT-One. It's all stored, and easily viewed.

Use it freely, and please post any improvements you make here, so others can enjoy it too. As I said, I'm not a programmer, and I have no doubt that things can be done better.
The UDP car/track names sent from the game is not coherent at all. Sometimes there's underscores, sometimes no spaces etc. This means that I have to manually add a line in the UDP program to change it. I have not gone through all cars and tracks, so if you find a strange name, please let me know and I'll fix it. Or maybe someone has a list of car names output by UDP?
Currently there's around 50 tracks and 70 cars in the database. I can add more, but the scraping currently takes around 2 hours. If you want, you can quite easily customize it to your liking. I can provide detailed steps on how to do it.

The future
If you are interested, we could look at running it all on Azure serverless. This way, console players could upload their data to the same database, also including PC players (from scraping). A Grand Unified Leaderboard! JasonSandwich, that has supported me in the development, knows more on this. Console players could use a Raspberry Pi, or small computer application, or an Android app, or something else, that receives the UDP and transmits data to the Azure server. I haven't looked into this yet, but it's an exciting thought. And fairly cheap (psst SMS :))


Use at your own risk. I take no responsibility. Have fun!

Maskmagog
13-01-2019, 13:38
Update 190511: This post is obsolete, and only useful if you want your own local database. Otherwise, all you need is the small Windows program. Get it at https://projectcalc.org/download.php.




---------------------------------------------------
What you need:
Raspberry Pi. There's several variations. I run it on a Raspberry Pi 3.
Power supply. Around 2-2.5A. Depends slightly on the version of RPi. Most places that sell the pi sells appropiate power supplys too. An old phone charger might work.
Case for the Pi (optional).
Micro SDHC card. I suggest 32GB or bigger. The image is done from a 16GB card, but if you're unlucky, another manufacturers 16GB card might be slightly smaller.
Ethernet cable between RPi and router. Might work with WiFi too, but wired will probably be more stable.

A computer with a SDHC card reader/writer.
For Windows there's a program called Etcher for writing images onto SDHC cards. https://www.balena.io/etcher/

Grab the file from the link below, unpack it, and write the image file to your sdhc card, then insert it in your Raspberry Pi.

I *think* the rest will be automatic. The program should get your gamertag automatically from the UDP stream.

Find the ip of the Pi on your network.
Browse to <your-rpi-ip>/index.php. Hopefully you'll see the leaderboard! :)
Bottom left it link to settings. At the moment the program only accepts valid laps, set in timetrial. If you want to, you can change that here.

Link to image file, last updated 190212:
https://www.dropbox.com/s/i1wol7hvenqho7j/pcalc_190212.rar?dl=0



If you want to change stuff yourself on the Pi:
EDIT: SSH should already be active, so adding the ssh file should not be needed (Add an empty file called 'ssh' to the root of the sdhc card, while it's still in your computer. This activates SSH on the Pi.)
To login to the Raspberry pi, I use the program 'putty'. Username is 'pi'. PM me for password. You can later change password etc by entering 'sudo raspi-config' when logged in.

The MariaDB can be accessed by 'sudo mysql -p'. Pass PG3Dnq4m2BVFaaLC.

The scraper is in /home/pi/pc2scraper/pc2scraper.py, and can be edited via putty with the command 'sudo nano /home/pi/pc2scraper/pc2scraper.py', or (preferrably) on a regular computer. I use WinSCP to login and move files to/from the Pi.
Copy the cars/tracks you want from the bottompart of the python file (excluding the #) and paste into either customtracks or customvehicles (row 178-179).
Run the scraper with 'sudo python /home/pi/pc2scraper/pc2scraper.py'. With around 70 cars and 50 tracks, it takes around 2 hours. It's currently run automatically every tuesday at 01:30.

The UDP script, which is built with Visual Studio, is in /home/pi/pc2udp/pc2udp.exe. Should start automatically, as a service, and run in the background.
The command 'sudo systemctl status pc2udp' will show if the udp program is running in the background, and show the latest output.

All source files are here: https://github.com/Maskmagog/projectcalc

ian2726
14-01-2019, 20:58
why cant I be smarter!!!

Maskmagog
14-01-2019, 22:10
Once it is setup, you don't have to do anything. The Pi (hopefully :)) takes care of everything: storing your laps, updating laptimes from Steam leaderboard, updates the program, getting your gamertag etc. While the program is free to use, you need some hardware (Raspberry Pi, charger, sdhc card, short ethernet cable) but totally worth it to me, as I enjoy time trial. I run it on a Raspberry Pi 3, but it would probably run fine on an older Raspberry Pi 2, if you have one lying around.

Please feel free to ask any questions about it, or any ideas that come to mind. I'd be happy to provide detailed instructions on what to buy and how to set it up.

Hopefully we can evolve this into something really good!

Maskmagog
16-01-2019, 18:47
Just realized that the two UDP protocols is very similar. I'll try to make an UDP 1 version too. Anyone has some pointers for doing so?

Doing some updates now, so if you're interested in trying it, maybe hold off a few days to get the newer version. It's supposed to update itself, but haven't had time to test it yet..

Maskmagog
18-01-2019, 18:23
Well, the protocols were *not* as similar as I initially thought... But now I think I get all the data I need for the UDP script. So a PC1 UDP version is on the way, for anyone that prefers that format. I might go back to that format myself, cause I do miss my crew chief!

Maskmagog
22-01-2019, 17:04
UDP v1 protocol is now supported too. Updated files on github: https://github.com/Maskmagog/projectcalc
If anyone is interested, I can provide a detailed walkthrough.

We have started work on moving the database and web pages to Azure (when I say we I mean JasonSandwich :)). Very exciting to see if it works. This could be a way to get One Leaderboard to Rule Them All, pc and console players. A lot of if's on the way though.

JasonSandwich
23-01-2019, 03:47
Time permitting, this will actually give console users working leaderboards meshed with the public Steam leaderboards. It won't read your old times but new times will be uploaded to a database and visible to anyone. This is all ambitious but not impossible. Maskmagog set the first time today from his Pi to the cloud database.

Once it works we have plenty of opportunity for scaling and delivering different endpoints for different geographies.

Maskmagog
14-02-2019, 16:34
Small update, I've updated the download link in post #2.
Added a "Top times per car"-button. This shows the top time for each car on selected track, sorted by laptime. A good way to compare different cars at a given track.
Otherwise it works great for me. Going forward, I'll use the time and energy to get it to work on Azure, so all users can upload their times to the same database.

breeminator
15-03-2019, 18:30
Hi, i'm new to Project Cars 2, but just wanted to say this looks like really great work by you guys. I'm a software developer myself, though not really in areas that would let me help out with this, but I know enough to have an idea of how much work you'll have been putting in to make progress on this. Getting a single merged leaderboard that combines PC times with times from anyone using Project CALC with a console would be great.

Do you think it will be possible to transfer an entire local database to Azure, i.e. is it worth people starting to use this right now in the knowledge that all their data will eventually find its way into the one database to rule them all?

Maskmagog
15-03-2019, 20:43
Hi breeminator, thanks! I'm not a professional software developer, you can imagine how much time it has taken me :)

We have actually a test version on a server, https://projectcalc.org. It's not finished at all, it's on a slow server etc, but it seems to be working just fine, for me anyway. It's still in development, but if you want you can try it out. I have a small program that runs on a Windows PC, just a small console program that reads & interprets the udp feed, and then sends valid laps in Time Trial to the database.

Once the program is running, launch a browser and go to projectcalc.org, enter your gamertag on the settings page, then enter your platform/controller/normal setup used. From the leaderboard page, press Race Mode button, and hopefully you will see your name at the bottom, world record on top, and chosen number of rivals that's just ahead of you on the leaderboard. When you set a lap, improve your lap time, or change car/track, the page should pick that up and update accordingly.
The laptimes is of course recorded even if you don't browse to the site, as long as the PCalcUpdater.exe is running.

All cars and tracks aren't in the database. Let me know if you miss a certain car or track.
Start the exe before entering Time Trial. The UDP feed only sends info on username, car, track etc at certain points, so if you start the PCalcUpdater too late, it might take a while before it's sent again. The program will write a message to console when it's picked up. It will also write a bunch of other stuff, like sector times etc, and if a lap is sent to the db.

There's plenty of car and track naming errors in the udp feed. I have to correct them manually in the program. Some cars have no spaces in the names, some have underscores etc. Let me know if you find any weird names.

Always get latest version here: projectcalc.org/download.php (https://projectcalc.org/download.php)

Remember it's just a test version, but I'd really appreciate if you (or someone) wants to test it. I'm still testing stuff, so some buttons etc on the site might not do anything, etc etc :)
And don't be afraid to suggest improvements, I have no doubt that everything can be improved.

JasonSandwich
18-03-2019, 16:25
I want to make sure people know that all you need to do for your times to be recorded is:

1. Start the exe (and make sure it connects)
2. Make sure you're in UDP v1 instead of v2 within the game (you should only need to do this once)
3. Go to Time Trial and do some laps.

The web page makes it much more fun but the times will still be uploaded to the database so that you can choose to go back and check them out later. Race mode on the website is fantastic but to contribute times you only need to have the executable running. That's it!

Maskmagog
06-04-2019, 12:33
Update: Web site is much faster now, see https://projectcalc.org.

Always get latest version here: projectcalc.org/download.php (https://projectcalc.org/download.php)

If you enjoy time trial on console, I would love to hear what you think.

Maskmagog
03-05-2019, 19:24
New version that supports point-to-point tracks, grab it at projectcalc.org (https://projectcalc.org)

It's a small, lightweight program that runs on Windows, or for example a Raspberry Pi.

Maskmagog
16-05-2019, 11:27
New version, grab it at projectcalc.org/download.php

Maskmagog
21-05-2019, 08:59
Update 0.5.5.20: added check so it won't upload laps before track, vehicle etc is received. Also added info about new versions available. Update available at projectcalc.org/download.php (https://projectcalc.org/download.php)

Maskmagog
21-05-2019, 22:24
Update 0.5.5.21: Added app version to database, for easier error checking. Still issues with and sometimes it seems. Please update to the new version at projectcalc.org/download.php (https://projectcalc.org/download.php).

Maskmagog
22-05-2019, 22:23
Made some changes on the server side, and etc should work fine now.

psFiNCH
24-05-2019, 17:17
Hi Maskmogog,

first, I want to thank you for your great work with Project CALC,
I know how much work that was & I hope that you will continue to work on it in the future.

Now I have some questions for you :

For example I would like to know if there is the possibility to change the program settings ?

In the console window of "Project CALC" you can see that ...

Settings: Timetrialonly: Y and validlapsonly: Y
Replay? 0=no, 1=yes: 0

In my opinion, it should only be possible for the best times to come into the database,
which were also driven under time trials & so far it looks like it.

But it also looks as if there are somewhere settings options,
because where a "Y" is, could also be an "N" or right ?

If I understood you correctly, you can use Project CALC, for example, exclusively in private circles.
One operates the "Project CALC"-server, but does the lap time from the Steam Leaderboard also end up in the database or can one adjust that too ?

And my final question :

I've created a bookmark & if I click on it, I'll end up on the right page, but I'll have to click on "Show Leaderboard" to see the table.
I think it's because the necessary COOKIE has to be created first.

OK Maskmagog, again, you have done a great job & I hope that in the future more console players,
like me with my PS4 use your "Project CALC".

Have a nice evening and a great weekend.

:) psFiNCH

Maskmagog
24-05-2019, 23:20
Hi psFiNCH!
Thank you, it makes me happy that you find it useful! I'm just a hobby programmer, but I really enjoy this and will hopefully be able to support and develop this further.


Regarding the settings: They are a leftover from previous versions. At first, I ran everything on a local Raspberry Pi. I then had an option to allow all laps, or just laps in time trial, and to allow only valid laps or invalid laps too. When I moved everything to a server, I deleted those options, and only allow valid laps set in time trial mode.
The replay settings output is also a leftover from testing. I discovered that the program had difficulties separating "live" laps from replays.
I will delete the console output regarding these things, as they are now obsolete. It is not possible to change them.

Regarding private circles, I am working on the possibilty to setup private leagues. Can't promise when it will be ready, but hopefully soon. Most of the work is done, just need to find the time for the last bits, and for some testing. It will be a no-frills solution, no passwords etc, to make it as easy as possible. I'll post here when there's a test version! The times set in a private league will also be inserted in the 'main' leaderboard.

The last question is something I can't answer actually :) It might be the cookie, it might be the way parameters are passed to the php, I'm not sure. It is a bit annoying, if anyone has a suggestion please let me know!

Again, thanks! Yes, hopefully more console players find this useful, please spread the word! And when Project CARS 3 comes out, I'll be ready :)

Maskmagog
25-05-2019, 11:31
Updated version: 0.0.5.24
Added private leagues.
Download at projectcalc.org/download.php (https://projectcalc.org/download.php)

First, you must go to projectcalc.org/settings.php (https://projectcalc.org/settings.php). Enter gamertag, set platform, controller, then you can either create or join a league.
Only laps done after this will register in the league.
On leaderboard page, click "Filters" and select league name, then Show leaderboard to show only that leagues lap times (for selected car - track of course).
Not fully tested, so please try it, and leave feedback!

JasonSandwich
26-05-2019, 03:01
The idea of cross-platform leagues is very exciting. How will this work for the Steam users? Will they just register for the league with their Steam name on this page and let the Steam leaderboards do the work or is this only for the console users who use the UDP tool?

Maskmagog
26-05-2019, 12:09
Good question, haven't given Steam users that much thought tbh. They could wait for the site to download Steam leaderboards, but that wouldn't be much fun I think. That might take up to a week. (It is possible to force updates to any leaderboard though, by adding &update=y to the url.)

I could change the database indexes. Then Steam users could run the PCalcUpdater tool too, and get their times in the database in realtime, like console users. The indexes would keep it from accepting double entries. The downside is that if people change username, the same lap might appear several times in the db. There might be a way to clean out those though, not sure.

I'm obviously all for cross-platform stuff, so I'll probably make some changes to the database indexes. Then Steam users can join the fun too, in the same way console master race enjoy it :)

Maskmagog
26-05-2019, 17:04
Ok, I've edited the database indexes. Steam players can also download PCalcUpdater (from here (https://projectcalc.org/download.php)) and use it just like console users. Their laptimes will instantly be inserted into the database, and they can use the Race Mode screen too, to follow their way up the leaderboard.

Just like console players, first go to projectcalc.org/settings.php (https://projectcalc.org/settings.php) and enter your Steam username, and set your preferences (platform, controller etc).

So, cross-platform leagues is on! As usual, I'd love to hear some feedback, good or bad. :)

Maskmagog
26-05-2019, 21:52
Just realized that I made an error when adding a simple check in the php scripts, which made the server not accept any laps. I forgot a $ sign at one place. Sorry if you have tried it over the weekend! Fixing the error now.

Maskmagog
26-05-2019, 22:21
Problem fixed, apologies if you have tried it this weekend and it didn't work.
psFiNCH, I see in the database that a few of your friends (I guess, since you have chosen the same combo) have tried it lately. Please let them know that it should work properly now.
Again sorry.

Maskmagog
29-05-2019, 19:34
New version: 0.5.5.28
Added some more stuff regarding private leagues. Fixed a track name. Various stuff added :)
Download at projectcalc.org/download.php (https://projectcalc.org/download.php)

Maskmagog
08-06-2019, 12:49
New version: 0.5.6.08
Various fixes such as track and vehicle names. Various stuff added.
If you use an older version, please update, otherwise it might not work properly.

Download at projectcalc.org/download.php (https://projectcalc.org/download.php)

Maskmagog
17-06-2019, 20:54
New version: 0.5.6.17
Changed method for identifying tracks. This solves issues with localized track name translations.

Download at projectcalc.org/download.php (https://projectcalc.org/download.php)