Instead of relying on people to remember to run a script to regenerate
it, just do it automatically at load time. Only takes a couple seconds.
And hey, this gives us a free test too! (CI will complain if load+dump
isn't clean.)
Fixes#334Fixes#74
Python 3.6 deprecated using regex flag syntax (?x) to set flags in the
middle of a pattern. Now you can only use it at the start of a pattern.
Fortunately that same release added a new scoped-flag syntax, (?x:).
(Wait, you say, it looks like our code *does* set flags at the start of
the pattern? That's true, but the markdown module includes our regex in the
middle of a larger one, so it's not actually at the start.)
Fixes this warning (and a couple similar ones):
DeprecationWarning: Flags not at the start of the expression '^(.?)(?x) \[ ([^]]' (truncated)
self.compiled_re = re.compile(r"^(.?)%s(.)$" % pattern
Fixes the following warnings when run with Markdown 2.6.11:
DeprecationWarning: "safe_mode" is deprecated in Python-Markdown. Use an HTML sanitizer (like Bleach https://bleach.readthedocs.io/) if you are parsing untrusted markdown text. See the 2.6 release notes for more info
DeprecationWarning: Using short names for Markdown's builtin extensions is deprecated. Use the full path to the extension with Python's dot notation (eg: "markdown.extensions.extra" instead of "extra"). The current behavior will raise an error in version 2.7. See the Release Notes for Python-Markdown version 2.6 for more info.
Also get rid of some compatability code for Markdown 2.0, which we no longer support.
Updates #257
Legendary and Mythical are official terms which, while generally not
present in the game data, are more or less consistently applied in
promotional material. They're also useful categorizations to fans,
so it seems reasonable to include them here.
The legendary Pokémon are:
Articuno, Zapdos, Moltres, Mewtwo,
Raikou, Entei, Suicune, Lugia, Ho-Oh,
Regirock, Regice, Registeel,
Latias, Latios, Kyogre, Groudon, Rayquaza,
Uxie, Mesprit, Azelf, Dialga, Palkia, Giratina,
Heatran, Regigigas, Cresselia,
Cobalion, Terrakion, Virizion,
Tornadus, Thundurus, Landorus,
Reshiram, Zekrom, Kyurem,
Xerneas, Yveltal, Zygarde,
Tapu Koko, Tapu Lele, Tapu Bulu, Tapu Fini,
Cosmog, Cosmoem, Solgaleo, Lunala, Necrozma.
The mythical Pokémon are: Mew, Celebi, Jirachi, Deoxys, Phione, Manaphy,
Darkrai, Shaymin, Arceus, Victini, Keldeo, Meloetta, Genesect,
Diancie, Hoopa, Volcanion, Magearna, Marshadow, Zeraora.
- Add exception handling for SQLite, PostgreSQL, and MySQL.
Sqlalchemy gives different exceptions for each database engine so I have
seperated each exception for each engine.
The error message passed by each engine is also different so that is
checked as well.
- Once the function gets one of those exceptions the following message
is outputed and the program ends: `ERROR: Table 'TABLENAME' already
exists in the database. Did you mean to use 'pokedex load -D'`
- If the error is not that a table already exists, then the output will
just be the full error from sqlalchemy. That way if someone runs into
that error it can be reported as an issue and whoever can help would see
the full error.
---
Resolves part of issue #16
Roserade and Roselia were mistakenly sharing dex number 72. Presumably
an error from the original X/Y spreadsheet. I guess we never reripped
these after 3DS rom dumping became possible.
Regardless of how it happened, add a test and some database constraints
to ensure that it doesn't happen again.
- Add a test for gaps in pokedex numbers
- Add uniqueness constraints to pokemon_dex_numbers. A species can
only appear once per pokedex, and a number cannot be used more than
once per pokedex.
Also require location area identifiers to be unique to their location.
I almost added a duplicate pokemon-league location when adding S/M
locations. This should help.
I'm going to record gift-pokemon as having no encounter slot, which I
believe is correct by-and-large (since you always get the same mon).
There are a few randomized gift pokemon which do have some type
of encounter slot (eg. Crystal's Odd Egg). Those aren't tracked (yet).
Starting in Gen VI, locations have been able to have an optional
subtitle. In-game, the subtitle is displayed on a second line under the
main name.
In X/Y this is used to give some locations an alternate,
flavorful name; for example, Kalos Route 2 is also known as Avance
Trail. In S/M this is mainly used to divide some large locations like
Hau'oli City into separate sections: e.g., the Beachfront, the Shopping
District, and the Marina.
This commit just adds the column; there are no data changes.
I suppose we'll need to go back and re-rip X/Y location names now.
When you open a file in Python 3, it defaults to using the system charset to encode the file, which is typically UTF-8 on linux systems (good) but Windows-1251 on Windows (bad).
We need to add explicit encoding=utf-8 arguments to open() calls when we open CSV files for reading or writing. To complicate matters, the csv module works only with byte strings in Python 2, and only with unicode strings in Python 3, so we can't just blindly use `io.open` everywhere.
PyPy (>= 3.6) was consistently erroring out during `pokedex load`,
with the error
OperationalError: (_sqlite3.OperationalError) cannot commit transaction - SQL statements in progress (Background on this error at: http://sqlalche.me/e/e3q8)
It turns out PyPy was not garbage collecting something, causing a
database cursor to be left open. See the giant comment block for
details.
* SQLAlchemy 1.0 introduced "baked queries" - a way to construct Query
objects so that they can be cached and reused.
* SQLAlchemy 1.2 changed lazyloaded columns to use baked queries under the
hood.
* Our MultilangQuery class attempts to set _default_language_id right
before the query is executed by overriding the __iter__ method.
* Baked queries bypass the __iter__ method and call a lower-level
method, _execute_and_instances, directly.
* This caused problems where _default_language_id wouldn't get set
correctly on lazyloaded columns.
* To fix, make MultilangQuery override the _execute_and_instances
method instead of __iter__.
* This is really just a stopgap: the root cause is that query params
are not preserved across lazyloads.
Tested with SQLAlchemy 0.9.7, 1.1.18, and 1.2.5.
Updates #236.
Went though the table docs and noted which tables have ids that correspond
to game ids, and explained why we sometimes have ids in the 10000s.
Since docs attached to id columns aren't displayed in the web documentation,
these edits all go in the class docstring, and i took the liberty of deleting
the id column docs from every table i touched.
Also: added some words about how to get to the species from pokemon and
pokemon_form.
Also: note that item flags are not official (they are a holdover from
brownkun) and item pockets are sort of kind of official (they originated from
D/P i believe, but i'm not sure how we've been assigning them for newer items).
Pertains to #99.
COPY FROM FILE requires database superuser permissions,
because of the obvious security implications.
COPY FROM STDIN has no such restriction.
Also do some cleanup while we're here.
SQLAlchemy 1.0 sets bindpararms to their default values when loading
lazy-loaded columns. This is in contrast to the 0.9 behaviour of
ignoring our incongruous bindparam alltogether.
So MultilangQuery is still broken, but now it breaks in the same way as before.
While we're here, set the correct param in one of the multilang tests. Not that
it matters.
The Column class accepts a 'doc' argument. Use it.
And while we're at it, make them all unicode strings.
Performed by the following sed script:
s/info=dict(description=u\?\("[^"]*"\))/doc=u\1/
s/info=dict(description=u\?\('[^']*'\))/doc=u\1/
s/\(\s*\)info=dict(description=u\?\("[^"]*"\), /\1doc=u\2,\n\1info=dict(/
s/\(\s*\)info=dict(description=u\?\('[^']*'\), /\1doc=u\2,\n\1info=dict(/
/info=dict(description=u\?\('[^']*'\),$/ {
s//doc=u\1,/
n
s/^\s*/&info=dict(/
}
I don't really like this but ehhhhhhh, the system for prose
translations seems to be to keep them in csv/translations/, and I can't
figure out how you're supposed to DO that, plus judging by the age of
the single file that's in there, that seems to be where translations go
to die.
<Zhorken> I'd like to motion (again) that 0 and 1 power both be stored
as null
<Zhorken> and — or * be displayed based on damage class
<Zhorken> which is what determines 0 vs 1 anyway, with the arbitrary
who-fucking-cares exception of Me First
<eevee> the ayes have it
<Zhorken> awesome
<eevee> that's a good idea i don't remember its being motioned the
first time
<Zhorken> I definitely remember arguing it when B/W released
MySQL again. Indexed keys have a maximum length of 767 bytes and unicode
is stored as 3 bytes per character. Since 256*3 = 768, that pushes us
just past the limit. Could lower the length to 255 characters, but let's
go further - currently only names and identifers have a length limit,
and if a name or identifier doesn't fit in a terminal then i don't want
to see it.
Fixes issue #123.
