Fetch game metadatas (images, ...) with external database

[samuelmichaud]

Hi,

Might be a little bit related to this issue : #44

As a developer, I want to build a frontend based on games detected by GLC and stored on glc-games.json which display games metadatas (mostly images but why not rating, ...).

Proposed evolution

Add a command line support to launch a new process "fetchForGameMetadata"

This process would do the following:

1. Download the (free & opensource) whole database https://thegamesdb.net/ in json with this url : https://cdn.thegamesdb.net/json/database-latest.json (100MO)
2. (Optionnal) Optimize the database by filtering on "platform == 1" only to get only PC games and not other platforms (to go from 90k games to 18k games).
3. Iterate on each detected games locally and match with title (transformed into a slug for better match) from the database. If a match is found, add and store game metadata on glc-games.json

NB : the DB dump don't give image urls, however, we can guess it with ID.
God of War => id = 96733
Image => https://cdn.thegamesdb.net/images/thumb/boxart/front/96733-1.jpg
Based on this page https://thegamesdb.net/stats.php it will work 98.54% of the time (boxart-front).

Thanks a lot for your awesome work, you guys rocks! I've a small personnal project to build a light TV frontend to launch games which support game pad controller, touch inputs,... I could not do it without your works. FYI, not finished, need to be opensourced.

[Nutzzz]

Hi @samuelmichaud . Welcome!

As indicated in #44 , we're obviously all for implementing this kind of feature.

A few notes:

1. Per Migrate games to database solution #6 , in the new_glc branch @Solaire is moving to a SQLite database for the backend, so I don't think we'll be adding metadata to the current json-based master branch. Admittedly his progress has been rather slow, but I wanted to warn you to the extent this may affect your plans for your frontend (though pulling in SQL data isn't particularly more difficult than json, and there are examples in some of the platform implementations).

2. As may be clear from my work on this project if you've been following along, I often err on the side of the quick and dirty solutions--so this is totally my kind of idea ;-) --but from what I can gather of thegamesdb.net's (admittedly informal) terms of service, In the interest of lessening their hosting workload, apparently the proper way to leverage their database would be to distribute it ourselves and/or request a key, and use the API documented here for updates.

3. Given that, here's my suggestion to tweak your proposal: since the data is apparently free (even for commercial use), we could download thegamedb.net's database based on the latest version at the time of a new GLC release, filter it for our purposes as you suggest, then (optionally) distribute the ~20MB file here. Better yet, we could get a smaller file by converting the filtered data to a binary format, e.g., a database file (note a .sql file is also available from thegamesdb.net to create a SQL database based on the latest data, though as it's designed for MySQL, it would need some massaging to create a SQLite version). Then, either way, use their API to do any updates at the user's end. This should also help avoid hitting their API quota, even if this niche project suddenly got a lot more popular.

In any case, I'm very interested to see what you've put together with your frontend whenever you're ready to share.

EDIT: Discovered there's an unofficial API wrapper that should make things easier.

[Nutzzz]

FWIW, I've requested and been granted an API key for TheGamesDb.net. Assuming I continue to have time (work's been slow), I'll do some playing around.

I've also been checking the various launcher databases we're already importing to see what metadata is already included. As might be expected, It's hit-and-miss.

[samuelmichaud]

Hi,

Thanks for your anwser & interest !

FWIW, I've finaly used https://rawg.io/ as primary source, more than TheGamesDb.net because images from TGDB are not always on the exact same ratio.

For non commercial / hobby project, it's max 20 000 api calls per month but I didn't see anything about dump in their terms : https://rawg.io/apidocs#terms so maybe it's allowed.

In any case your suggestion in 3) seems perfect.

Also, I've seen that Pegasus frontend (https://github.com/mmatyas/pegasus-frontend) scan medias in folders of detected games. Maybe this is something you can explore too...
They also seems to use other sources but I couln't figure which ones yet.

[Nutzzz]

Yes, consistent image ratio is obviously preferable, but the RAWG terms, while not exactly onerous, do make things a bit more difficult in this context:

1. With a console app, we're dealing with limited screen real estate, so finding a spot for attribution to them on "every page" where their data is displayed isn't as easy as one might think.
2. I'm also not sure about an "active" hyperlink implementation that they require for attribution. We've talked about adding mouse support in issues C++ re-implementation #11 and GLC 2.0 #39 , but it's a lower priority, and I'm not sure I want to make it a higher priority just for this. I guess we could assign a key and use that key to open the default web browser to the appropriate link, but then it should probably be clear which key that is, and assuming that key is configurable you go from 4 characters to eight for something like "[W] RAWG". Anyway, less than ideal.
3. I think using a dump is out per "no data redistribution."

I did a few searches of Pegasus' codebase and I can't really see anything very helpful. Let me know if there's a specific bit of code you think I missed.

[Nutzzz]

Since this is all part of GLC2, we probably want to implement metadata providers as plugins like the platforms. Here's some we might start with:

• Giant Bomb
• IGDB
• MobyGames
• RAWG
• TheGamesDB

[samuelmichaud]

Hi,

I told you I will opensource my work... it took a very long time but here it is : https://github.com/samuelmichaud/PhotonLauncher

Thanks guys, I could not have done it without your work !