Skip Menu |

This queue is for tickets about the DBIx-Class-Schema-Loader CPAN distribution.

Report information
The Basics
Id: 74170
Status: resolved
Priority: 0/
Queue: DBIx-Class-Schema-Loader

People
Owner: Nobody in particular
Requestors: PCMANTZ [...] cpan.org
Cc:
AdminCc:

Bug Information
Severity: Important
Broken in:
  • 0.07009
  • 0.07015
Fixed in: (no value)

Attachments


Subject: Data types not picked up for Sybase ASE 15.5
A couple months ago, we migrated our server from 12.1 to 15.5, after switching our ORM from something hand-rolled to DBIx::Class. Our workflow for schema updates starts with making changes to the development environment, running DBIC::S::L on the database, and committing the ensuing changes (including some post-checksum massaging) to our repository. So far, this has kept our ORM in lock-step across all environments, and even allows us to use SQLite as a testing database. I noticed that after the upgrade, the DBIC::S::L component has been dropping datatypes from our Result classes. I cracked open DBIC:S:L:DBI:Sybase and found the following query is being used to retrieve column information, and created a test table: --- create table foo ( id int identity, bar varchar(20), baz datetime default getdate(), constraint PK_foo primary key clustered (id) ) --- I then interpolated the requisite values from the query used in _columns_info_for to see what I would get: --- SELECT c.name name, bt.name base_type, ut.name user_type, cm.text deflt, c.prec prec, c.scale scale, c.length len FROM syscolumns c JOIN sysobjects o ON c.id = o.id LEFT JOIN systypes bt ON c.type = bt.type LEFT JOIN systypes ut ON c.usertype = ut.usertype LEFT JOIN syscomments cm ON cm.id = CASE WHEN c.cdefault = 0 THEN c.computedcol ELSE c.cdefault END WHERE o.name = 'foo' AND o.uid = 1 AND o.type IN ('U', 'V') --- name,name,name,text,prec,scale,length baz,datetime,datetime,DEFAULT getdate(),,,8 --- (NOTE: no renaming of columns here) This is troublesome! It appears that this query is only returning information for columns that have a default value. Not all of our columns have a default value, nor should it be expected. This means that they don't get an entry in the syscolumns table, so this really mucks up our ability to generate a schema. As an experiment, I removed the join with syscomments to see what would get returned: --- SELECT c.name name, bt.name base_type, ut.name user_type, c.prec prec, c.scale scale, c.length len FROM syscolumns c JOIN sysobjects o ON c.id = o.id LEFT JOIN systypes bt ON c.type = bt.type LEFT JOIN systypes ut ON c.usertype = ut.usertype WHERE o.name = 'foo' AND o.uid = 1 AND o.type IN ('U', 'V') --- name,name,name,prec,scale,length bar,sysname,varchar,,,20 bar,varchar,varchar,,,20 bar,nvarchar,varchar,,,20 bar,longsysname,varchar,,,20 id,int,int,,,4 baz,datetime,datetime,,,8 --- This is better, sort of. It shows that the type information is available for all columns, but you don't get the default info, since it is stored in syscomments. I have a few questions regarding the operation here: * Why is this query dropping the unmatching results on the left hand side of the query? Isn't the point of a left join that the left information is kept even when the right-hand information is missing? * Has anyone tested/confirmed DBIC:S:L on Sybase ASE 15.5? Is there any further testing I can do? * Are there any database configuration options regarding column comments that users should be aware of? * Would a two-pass approach (data-type, then default values) to information be preferable?
Subject: Re: [rt.cpan.org #74170] Data types not picked up for Sybase ASE 15.5
Date: Thu, 19 Jan 2012 18:55:54 -0500
To: bug-DBIx-Class-Schema-Loader [...] rt.cpan.org
From: Rafael Kitover <rkitover [...] cpan.org>
On Thursday, January 19, 2012 at 6:30 PM, Paul Custer Mantz via RT wrote: Show quoted text
> Thu Jan 19 18:30:30 2012: Request 74170 was acted upon. > Transaction: Ticket created by PCMANTZ > Queue: DBIx-Class-Schema-Loader > Subject: Data types not picked up for Sybase ASE 15.5 > Broken in: 0.07009, 0.07015 > Severity: Important > Owner: Nobody > Requestors: PCMANTZ@cpan.org (mailto:PCMANTZ@cpan.org) > Status: new > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=74170 > > > > A couple months ago, we migrated our server from 12.1 to 15.5, after > switching > our ORM from something hand-rolled to DBIx::Class. Our workflow for schema > updates starts with making changes to the development environment, running > DBIC::S::L on the database, and committing the ensuing changes > (including some > post-checksum massaging) to our repository. So far, this has kept our > ORM in > lock-step across all environments, and even allows us to use SQLite as a > testing database. > > I noticed that after the upgrade, the DBIC::S::L component has been dropping > datatypes from our Result classes. I cracked open DBIC:S:L:DBI:Sybase and > found the following query is being used to retrieve column information, and > created a test table: >
Apparently on my machine I have ASE 15.0, I didn't see this problem. If you open up t/10_06sybase_common.t in the distribution you will see the data_type tests, not all of the columns in those tests have defaults. Please run this test on your server and give me the output. Show quoted text
> > --- > > create table foo ( > id int identity, > bar varchar(20), > baz datetime default getdate(), > constraint PK_foo > primary key clustered (id) > ) > > --- > > I then interpolated the requisite values from the query used in > _columns_info_for to see what I would get: > > --- > > SELECT c.name (http://c.name) name, bt.name (http://bt.name) base_type, ut.name (http://ut.name) user_type, cm.text deflt, > c.prec prec, c.scale scale, c.length len > FROM syscolumns c > JOIN sysobjects o ON c.id = o.id > LEFT JOIN systypes bt ON c.type = bt.type > LEFT JOIN systypes ut ON c.usertype = ut.usertype > LEFT JOIN syscomments cm > ON cm.id = CASE WHEN c.cdefault = 0 THEN c.computedcol ELSE > c.cdefault END > WHERE o.name (http://o.name) = 'foo' > AND o.uid = 1 > AND o.type IN ('U', 'V') > > --- > name,name,name,text,prec,scale,length > baz,datetime,datetime,DEFAULT getdate(),,,8 > --- > > (NOTE: no renaming of columns here) > > This is troublesome! It appears that this query is only returning > information > for columns that have a default value. Not all of our columns have a > default > value, nor should it be expected. This means that they don't get an > entry in > the syscolumns table, so this really mucks up our ability to generate a > schema. > > As an experiment, I removed the join with syscomments to see what would get > returned: > > --- > > SELECT c.name (http://c.name) name, bt.name (http://bt.name) base_type, ut.name (http://ut.name) user_type, c.prec prec, > c.scale scale, c.length len > FROM syscolumns c > JOIN sysobjects o ON c.id = o.id > LEFT JOIN systypes bt ON c.type = bt.type > LEFT JOIN systypes ut ON c.usertype = ut.usertype > WHERE o.name (http://o.name) = 'foo' > AND o.uid = 1 > AND o.type IN ('U', 'V') > > --- > name,name,name,prec,scale,length > bar,sysname,varchar,,,20 > bar,varchar,varchar,,,20 > bar,nvarchar,varchar,,,20 > bar,longsysname,varchar,,,20 > id,int,int,,,4 > baz,datetime,datetime,,,8 > --- > > This is better, sort of. It shows that the type information is > available for > all columns, but you don't get the default info, since it is stored in > syscomments. > > I have a few questions regarding the operation here: > > * Why is this query dropping the unmatching results on the left hand side of > the query? Isn't the point of a left join that the left information > is kept > even when the right-hand information is missing? > >
That's a question for your DBAs or some Sybase experts, I have no clue :) That is indeed the point of a LEFT JOIN. Show quoted text
> > * Has anyone tested/confirmed DBIC:S:L on Sybase ASE 15.5? Is there any > further > testing I can do? > >
Please run the sybase common tests in t/10_06sybase_common.t and give me the output. Show quoted text
> > * Are there any database configuration options regarding column comments > that > users should be aware of? > >
Don't think so, another question for DBAs. Show quoted text
> > * Would a two-pass approach (data-type, then default values) to > information be > preferable? > >
I personally could care less how many passes it takes, sl is already slow and I have no problems making it even slower if needed :) If you can hack the code with a two-pass approach to get it to work for your server, send me the patch. Use the git repo: git clone git://git.shadowcat.co.uk/dbsrgits/DBIx-Class-Schema-Loader.git sl if you send me a patch, also send me the full name, IRC nickname and email address you prefer to be listed in the attribution. If you would prefer to have access to the repo to work on a branch, send your SSH public key and I will add you. Good luck!
RT-Send-CC: rkitover [...] cpan.org
Here is the output of the test script, including stdout messages. You'll notice that data_type is just not being picked up, and the failing tests 237-277 confirm it. The following is the version number of the database I tested against: Adaptive Server Enterprise/15.5/EBF 18657 SMP ESD#4/P/Sun_svr4/OS 5.8/asear155/2545/64-bit/FBO/Thu Jun 16 10:29:40 2011 I'll see what I can do about a two-pass solution to this; as it seems that there is a query being issued for each individual column anyway.
Subject: sybase_common_test_scrubbed.out
Download sybase_common_test_scrubbed.out
application/octet-stream 166.8k

Message body not shown because it is not plain text.

Subject: Re: [rt.cpan.org #74170] Data types not picked up for Sybase ASE 15.5
Date: Mon, 23 Jan 2012 15:02:43 -0500
To: bug-DBIx-Class-Schema-Loader [...] rt.cpan.org
From: Rafael Kitover <rkitover [...] cpan.org>
On Monday, January 23, 2012 at 2:35 PM, Paul Custer Mantz via RT wrote: Show quoted text
> Queue: DBIx-Class-Schema-Loader > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=74170 > > > Here is the output of the test script, including stdout messages. > You'll notice that data_type is just not being picked up, and the > failing tests 237-277 confirm it. > > The following is the version number of the database I tested against: > > Adaptive Server Enterprise/15.5/EBF 18657 SMP ESD#4/P/Sun_svr4/OS > 5.8/asear155/2545/64-bit/FBO/Thu Jun 16 10:29:40 2011 > > I'll see what I can do about a two-pass solution to this; as it seems > that there is a query being issued for each individual column anyway. > > > Attachments: > - sybase_common_test_scrubbed.out >
Apparently I have 12.5: DBMS_VER SQL Server/12.5/P/Generic/OS/1/OPT/Sat Jun 30 00:01:37 PDT 2001 I will update my copy of ASE on my linux laptop over the next few days. If you want to do any optimizations while you're in the _columns_info_for code fixing this, feel free, although I don't think it makes a huge amount of difference. Selects on system tables are pretty fast. I'm working on an Oracle issue right now, will wait until I fix that and your patch and then do a new release.
While I'm working on this, is there any reason that the 'inflate_datetime' is being set? The Sybase & MSSQL drivers are the only ones that set it. Seems like it should be taken out.
On Fri Jan 27 16:02:54 2012, PCMANTZ wrote: Show quoted text
> While I'm working on this, is there any reason that the > 'inflate_datetime' is being set? The Sybase & MSSQL drivers are the > only ones that set it. Seems like it should be taken out.
It is set to 0 for timestamp, because timestamp in Sybase is not a datetime type. This is to tell InflateColumn::DateTime to not try to inflate timestamps as datetimes. Please do not take it out.
https://github.com/pcmantz/dbic-sl/tree/sybase_ase_15.5 Here's a branch of my work to get DBIC:S:L:DBI:Sybase to work with Sybase 15.5. All the tests pass, and I've used it to regenerate my schema. I've also removed the inflate_datetime field setting in the commit on that branch. If you still have a Sybase 12.4 installation around, I would appreciate it if you could test this. I haven't done so yet myself, and I don't know if I'll have that opportunity.
When did ASE 15 come out? I just installed ASE on my windows box a few months ago and sp_server_info is showing 12.5... How are you reading the server version? If you do an sp_server_info in the row that says DBMS_VER, what does it say for you? I'm going to leave the ASE on my linux box alone and upgrade the ASE on my windows box to the 15.7 they have out now so I can test both. Thanks for the branch, going to play with it over the next few days.
So on my linux box, select @@version shows: Adaptive Server Enterprise/15.0.3/EBF 16555 ESD#1/P/Linux Intel/Linux 2.6.9-42.ELsmp i686/as e1503/2680/32-bit/FBO/Thu Mar 5 04:50:21 2009 On my windows box, select @@version shows: Adaptive Server Enterprise/15.5/EBF 18158 SMP ESD#2/P/X64/Windows Server/asear155/2514/64-bi t/OPT/Wed Aug 25 05:39:57 2010 and I am about to upgrade the windows box to 15.7. Running the common tests on the 15.5 ASE on the windows box right now, but getting disconnects from the server in the middle of the tests for some reason...will investigate. IIRC when I ran them before, the data_type tests ran fine, I don't think this is specifically a version issue but I will verify shortly.
So the new developer's versions of ASE do not allow setting 'number of user connections' above 25. This breaks the sl common tests. I've tried both sp_configure and editing the .cfg to no effect. I'm going to work on reducing the number of active cursors in the common tests so it works with the 25 limit so I can test with my copy of the 15.5 developer edition.
On Fri Jan 27 23:13:51 2012, RKITOVER wrote: Show quoted text
> When did ASE 15 come out? I just installed ASE on my windows box a few > months ago and sp_server_info is showing 12.5... >
According to wikipedia, it came out in 2005: http://en.wikipedia.org/wiki/Adaptive_Server_Enterprise Though I can imagine that most shops were in no real hurry to upgrade, so there hasn't been much demand for support. Show quoted text
> How are you reading the server version? >
I was checking the database info using Squirrel SQL. Show quoted text
> If you do an sp_server_info in the row that says DBMS_VER, what does it > say for you? >
I don't have access to this database anymore, though I can ask some people that do to provide that info. Show quoted text
> I'm going to leave the ASE on my linux box alone and upgrade the ASE on > my windows box to the 15.7 they have out now so I can test both. > > Thanks for the branch, going to play with it over the next few days.
I finally got a chance to test on a new version of ASE developer's edition. I used 15.7 which I believe was just released. I had to reduce the number of active statements the common tests use, which turned out to be much simpler than I anticipated. All tests pass using the existing code, including the data_type tests, except for the nchar/nvarchar types, because I used a utf8 encoding, but I fixed that (new release coming shortly.) So I think the reason you were having this problem with your database was due to some sort of misconfiguration, or maybe 15.5 was just a bad version (unlikely) in any case, the existing code is fine. Your fork changes ALOT of code, spread across a few commits, so I'm not comfortable merging it as is, as there seems to be no reason to. I'd be more comfortable merging it if I knew the reason why it was needed, how this server is different from my 3 servers and that of the other users of this module who have not reported this problem, which would involve some more investigation. But you said you lost access to this server, so that may be difficult. I don't want your work to get lost though, so I'll pull it into a branch and test it out with my servers, perhaps add a config option to enable it or something, if you're interested.
I looked over your code again and it seems harmless enough, and maybe there are other users with this weird problem, so I made some changes and merged it to master. Will be releasing in a few minutes.
Also tested it on two of my ASE servers, the 15.0.3 and the 15.7, and it works perfectly!
Your branch is merged with some changes in 0.07022, closing this ticket.