UrlPool¶

class UrlPool(name, urls, *, strip_slash=True, ttl=3600)[source]¶

Pool of possible urls which provides easy access to a working one.

name¶

Name given to the pool

urls¶

List of the possible urls

strip_slash¶

Whether or not tailing slashes should be removed

ttl¶

Time until the current url expires

coroutine fetch(self)[source]¶

Get the current url from the database.

name

needs_update¶

Whether the current url is outdated.

prepare_url(url)[source]¶

Prepare an url to be used as the current url.

This function is performed for all urls returned by url

coroutine update_url(self)[source]¶

Search for a working url.

This is automatically called.

coroutine upload(self)[source]¶

Upload the current url to the database.

url¶

Current url.

Grobber