As part of the dapp I’m developing, I’d like to be able to maintain a list of users that are actually using the dapp server-side. Moreover, my intent is to make the server API configurable – anyone will be able to run their own server and point the dapp at it if they don’t trust or can’t access the public server. That means in order to maintain a list of users, the server needs to iterate through all possible users’ profiles to compile the list of actual users.
This approach was originally discussed in this post.
My question is two-fold:
From the perspective of my dapp, are all names and subdomains potential users whose profile I need to process and possibly add? There are currently about 1,350 pages of subdomains (or 135,000 subdomains) returned by the API. If I could whitelist/blacklist particular TLDs or subdomains, it would somewhat speed things up.
Has anyone dealt with doing indexing like this, and if so, what have you done to improve performance? As it stands, the current process requires thousands of requests to both the Blockstack core node and the Gaia hub. Running a core node locally helps considerably, but fetching the actual profile data from Gaia is still pretty slow due to rate limiting.
EDIT: After some additional thought, I think federation between the different servers may be an appropriate compromise:
- Each server would maintain a list of peers.
- A peer could be bootstrapped by either getting a list of usernames from an existing peer or by going through the process described above.
- When a user registers, an event would be broadcast to all peers to add the username.
The downside is that there’s now a level of trust required in assuming that the events are in fact being fired. On the other hand, this would also provide a mechanism for broadcasting other events, (like the user adding content) which would lessen the need to poll Gaia for that data as well.
I would still love to hear if anyone has additional thoughts around any of this though!