PDA

View Full Version : Project CALC - car/track leaderboards for consoles



Maskmagog
13-01-2019, 13:36
UPDATE 190406: Web site updated, see https://projectcalc.org.
Also slightly updated exe, download here: https://www.dropbox.com/s/xvlyvxd1qp....3.29.zip?dl=0
Unzip, run exe, do clean laps in timetrial.
If you enjoy time trial on console, I would love to hear what you think.



TL;DR: Run a Raspberry Pi (very small, cheap computer) that stores your laptimes, fetches PC players laptimes, and displays it all in any browser. Leaderboards per car/track combo for Xbox and PS4!

We all know the sad state of console leaderboards. So with some great help from others, I've built a solution that I call Project CALC – Community Assisted Leaderboards for Consoles. It runs on a Raspberry Pi, and saves all your laps (laptime, sector times etc).

Once the Pi is running, you don't have to do anything. The Pi automatically saves your laps. You can view the leaderboards from any browser. Race Mode changes automatically when you switch car or track. Just leave the Pi running.

This runs on the Pi:

* program that receives UDP data
* database where laps are stored
* Apache2 web server and php pages that displays everything
* Python script that scrapes the cars2-stats-steam page and stores the times in the database

It can use both the old PC1 and the new PC2 udp format.

The program receives and interprets various UDP data from the game. Thank you Zeratall for the library, and all the help! It checks for your current lap record for the car-track combo you are currently running. You could set it to record all laps, or just laps that are valid, or just laps in Time Trial, or whatever you want. Store the data you want. Track temp? Ride height? Tyre temps? Rain density?

The database stores the data, and you could use that data anyway you want. It's a MariaDB, very similar to MySQL, and easy to use.

The php page shows the leaderboards, with dropdown selections for track, car and class. It's also possible to show just your top times for each car for selected track, or show all your stored laps.

263504

263507

There's also a Race Mode, a php page that displays the WR, your best laptime, and the rivals just ahead of you on the leaderboard. All in big LCD letters. This updates automatically if you set a better laptime, or change car or track. I run this on an old tablet next to my monitor, it's really awesome, imho!

263506

This is a huge improvement over in-game leaderboards to me. I'm not a programmer at all, so it has taken a lot of trial and error. I know that there are plenty of computer programs that does this too, probably in a better way. The thing for me is that I don't want my computer turned on all the time, running the program. The Pi on the other hand, can be on all the time. It's slightly larger than a credit card, and costs about $35.

Including data from cars2-stat-steam
I, as many others, have used the cars2-stat-steam page to compare lap times. I found a scraper for this (Thanks Jonas Gulle!), and modified it to scrape all leaderboard pages that I want, and store them in the database. Now i can instantly see where I stand on my tablet: my laptimes are mixed in with the steam laptimes.

For me, the statistics are a big part of racing/gaming. I want to see my lap times and sector times, to analyze it. Now I can! No longer is my (for me) super awesome lap in a Ginetta G40 Junior erased by a crappy lap in a Toyota GT-One. It's all stored, and easily viewed.

Use it freely, and please post any improvements you make here, so others can enjoy it too. As I said, I'm not a programmer, and I have no doubt that things can be done better.
The UDP car/track names sent from the game is not coherent at all. Sometimes there's underscores, sometimes no spaces etc. This means that I have to manually add a line in the UDP program to change it. I have not gone through all cars and tracks, so if you find a strange name, please let me know and I'll fix it. Or maybe someone has a list of car names output by UDP?
Currently there's around 50 tracks and 70 cars in the database. I can add more, but the scraping currently takes around 2 hours. If you want, you can quite easily customize it to your liking. I can provide detailed steps on how to do it.

The future
If you are interested, we could look at running it all on Azure serverless. This way, console players could upload their data to the same database, also including PC players (from scraping). A Grand Unified Leaderboard! JasonSandwich, that has supported me in the development, knows more on this. Console players could use a Raspberry Pi, or small computer application, or an Android app, or something else, that receives the UDP and transmits data to the Azure server. I haven't looked into this yet, but it's an exciting thought. And fairly cheap (psst SMS :))


Use at your own risk. I take no responsibility. Have fun!

Maskmagog
13-01-2019, 13:38
What you need:
Raspberry Pi. There's several variations. I run it on a Raspberry Pi 3.
Power supply. Around 2-2.5A. Depends slightly on the version of RPi. Most places that sell the pi sells appropiate power supplys too. An old phone charger might work.
Case for the Pi (optional).
Micro SDHC card. I suggest 32GB or bigger. The image is done from a 16GB card, but if you're unlucky, another manufacturers 16GB card might be slightly smaller.
Ethernet cable between RPi and router. Might work with WiFi too, but wired will probably be more stable.

A computer with a SDHC card reader/writer.
For Windows there's a program called Etcher for writing images onto SDHC cards. https://www.balena.io/etcher/

Grab the file from the link below, unpack it, and write the image file to your sdhc card, then insert it in your Raspberry Pi.

I *think* the rest will be automatic. The program should get your gamertag automatically from the UDP stream.

Find the ip of the Pi on your network.
Browse to <your-rpi-ip>/index.php. Hopefully you'll see the leaderboard! :)
Bottom left it link to settings. At the moment the program only accepts valid laps, set in timetrial. If you want to, you can change that here.

Link to image file, last updated 190212:
https://www.dropbox.com/s/i1wol7hvenqho7j/pcalc_190212.rar?dl=0



If you want to change stuff yourself on the Pi:
EDIT: SSH should already be active, so adding the ssh file should not be needed (Add an empty file called 'ssh' to the root of the sdhc card, while it's still in your computer. This activates SSH on the Pi.)
To login to the Raspberry pi, I use the program 'putty'. Username is 'pi'. PM me for password. You can later change password etc by entering 'sudo raspi-config' when logged in.

The MariaDB can be accessed by 'sudo mysql -p'. Pass PG3Dnq4m2BVFaaLC.

The scraper is in /home/pi/pc2scraper/pc2scraper.py, and can be edited via putty with the command 'sudo nano /home/pi/pc2scraper/pc2scraper.py', or (preferrably) on a regular computer. I use WinSCP to login and move files to/from the Pi.
Copy the cars/tracks you want from the bottompart of the python file (excluding the #) and paste into either customtracks or customvehicles (row 178-179).
Run the scraper with 'sudo python /home/pi/pc2scraper/pc2scraper.py'. With around 70 cars and 50 tracks, it takes around 2 hours. It's currently run automatically every tuesday at 01:30.

The UDP script, which is built with Visual Studio, is in /home/pi/pc2udp/pc2udp.exe. Should start automatically, as a service, and run in the background.
The command 'sudo systemctl status pc2udp' will show if the udp program is running in the background, and show the latest output.

All source files are here: https://github.com/Maskmagog/projectcalc

ian2726
14-01-2019, 20:58
why cant I be smarter!!!

Maskmagog
14-01-2019, 22:10
Once it is setup, you don't have to do anything. The Pi (hopefully :)) takes care of everything: storing your laps, updating laptimes from Steam leaderboard, updates the program, getting your gamertag etc. While the program is free to use, you need some hardware (Raspberry Pi, charger, sdhc card, short ethernet cable) but totally worth it to me, as I enjoy time trial. I run it on a Raspberry Pi 3, but it would probably run fine on an older Raspberry Pi 2, if you have one lying around.

Please feel free to ask any questions about it, or any ideas that come to mind. I'd be happy to provide detailed instructions on what to buy and how to set it up.

Hopefully we can evolve this into something really good!

Maskmagog
16-01-2019, 18:47
Just realized that the two UDP protocols is very similar. I'll try to make an UDP 1 version too. Anyone has some pointers for doing so?

Doing some updates now, so if you're interested in trying it, maybe hold off a few days to get the newer version. It's supposed to update itself, but haven't had time to test it yet..

Maskmagog
18-01-2019, 18:23
Well, the protocols were *not* as similar as I initially thought... But now I think I get all the data I need for the UDP script. So a PC1 UDP version is on the way, for anyone that prefers that format. I might go back to that format myself, cause I do miss my crew chief!

Maskmagog
22-01-2019, 17:04
UDP v1 protocol is now supported too. Updated files on github: https://github.com/Maskmagog/projectcalc
If anyone is interested, I can provide a detailed walkthrough.

We have started work on moving the database and web pages to Azure (when I say we I mean JasonSandwich :)). Very exciting to see if it works. This could be a way to get One Leaderboard to Rule Them All, pc and console players. A lot of if's on the way though.

JasonSandwich
23-01-2019, 03:47
Time permitting, this will actually give console users working leaderboards meshed with the public Steam leaderboards. It won't read your old times but new times will be uploaded to a database and visible to anyone. This is all ambitious but not impossible. Maskmagog set the first time today from his Pi to the cloud database.

Once it works we have plenty of opportunity for scaling and delivering different endpoints for different geographies.

Maskmagog
14-02-2019, 16:34
Small update, I've updated the download link in post #2.
Added a "Top times per car"-button. This shows the top time for each car on selected track, sorted by laptime. A good way to compare different cars at a given track.
Otherwise it works great for me. Going forward, I'll use the time and energy to get it to work on Azure, so all users can upload their times to the same database.

breeminator
15-03-2019, 18:30
Hi, i'm new to Project Cars 2, but just wanted to say this looks like really great work by you guys. I'm a software developer myself, though not really in areas that would let me help out with this, but I know enough to have an idea of how much work you'll have been putting in to make progress on this. Getting a single merged leaderboard that combines PC times with times from anyone using Project CALC with a console would be great.

Do you think it will be possible to transfer an entire local database to Azure, i.e. is it worth people starting to use this right now in the knowledge that all their data will eventually find its way into the one database to rule them all?

Maskmagog
15-03-2019, 20:43
Hi breeminator, thanks! I'm not a professional software developer, you can imagine how much time it has taken me :)

We have actually a test version on a server, https://projectcalc.org. It's not finished at all, it's on a slow server etc, but it seems to be working just fine, for me anyway. It's still in development, but if you want you can try it out. I have a small program that runs on a Windows PC, just a small console program that reads & interprets the udp feed, and then sends valid laps in Time Trial to the database.

Once the program is running, launch a browser and go to projectcalc.org, enter your gamertag on the settings page, then enter your platform/controller/normal setup used. From the leaderboard page, press Race Mode button, and hopefully you will see your name at the bottom, world record on top, and chosen number of rivals that's just ahead of you on the leaderboard. When you set a lap, improve your lap time, or change car/track, the page should pick that up and update accordingly.
The laptimes is of course recorded even if you don't browse to the site, as long as the PCalcUpdater.exe is running.

All cars and tracks aren't in the database. Let me know if you miss a certain car or track.
Start the exe before entering Time Trial. The UDP feed only sends info on username, car, track etc at certain points, so if you start the PCalcUpdater too late, it might take a while before it's sent again. The program will write a message to console when it's picked up. It will also write a bunch of other stuff, like sector times etc, and if a lap is sent to the db.

There's plenty of car and track naming errors in the udp feed. I have to correct them manually in the program. Some cars have no spaces in the names, some have underscores etc. Let me know if you find any weird names.

Here is the program: https://www.dropbox.com/s/i5bhheht0m1lmkv/PCalcUpdater_190316.zip?dl=0
Unzip and run PCalcUpdater.exe (allow network access).

Remember it's just a test version, but I'd really appreciate if you (or someone) wants to test it. I'm still testing stuff, so some buttons etc on the site might not do anything, etc etc :)
And don't be afraid to suggest improvements, I have no doubt that everything can be improved.

JasonSandwich
18-03-2019, 16:25
I want to make sure people know that all you need to do for your times to be recorded is:

1. Start the exe (and make sure it connects)
2. Make sure you're in UDP v1 instead of v2 within the game (you should only need to do this once)
3. Go to Time Trial and do some laps.

The web page makes it much more fun but the times will still be uploaded to the database so that you can choose to go back and check them out later. Race mode on the website is fantastic but to contribute times you only need to have the executable running. That's it!

Maskmagog
06-04-2019, 12:33
Update: Web site is much faster now, see https://projectcalc.org.

Also slightly updated exe, download here: https://www.dropbox.com/s/xvlyvxd1qpe523z/PCalcUpdater_0.5.3.29.zip?dl=0

If you enjoy time trial on console, I would love to hear what you think.