Comments (14)
Hi @markddrake very strange. The way streams work are typically managed by Node.js itself. If this is something that this module needs to implement, would you be willing to make a pull request with a fix? I would expect from the Node.js docs that it would have at least some value, as this module will .push
a row in to the stream as it parses it from the query.
from mysql.
I did a quick hunt (30secs) to look for the .stream() implementation but didn't see it. I was really trying to see if / how destroy() / _destroy() was implemented. If you have some pointers on where to look.
from mysql.
No problem. Here is a direct link to the implementation:
mysql/lib/protocol/sequences/Query.js
Line 192 in 3430c51
from mysql.
I wonder if this is a side effect of
var Readable = require('readable-stream');
rather than (switching to modern practice)
import { Readable } from 'stream'
Note sure which version of 'stream' the 'readable-stream' module is mimicking
readable.readable length was Added in: v9.4.0
I also don't see an implementation of _destroy() (introduced in v8.0.0)
I was not aware of the issues around stream that readable stream was meant to address. I wonder if it's as simple as switching the source of the 'Readable' class.
from mysql.
It is just a copy in user land from my understanding, allowing to use the newer stream implementations in older Node.js versions (i.e. backport functionality). Does changing it fix the issue is the real question, though.
from mysql.
More importantly, does changing it fix the issue without breaking a 's**tload' of other things :)
from mysql.
A Quick Hack says yes, it does fix the issue with readableLength
Given the following code
this.yadamuLogger.trace([this.constructor.name,'PIPELINE ERROR',this.tableName,this.COPY_METRICS.SOURCE_DATABASE_VENDOR,this.COPY_METRICS.TARGET_DATABASE_VENDOR],`${
this.currentPipeline.map((s) => { return `${s.constructor.name}:[${s.readableLength},${s.writableLength}]` }).join(',')
}`)
I get the following output with the current implementation
[TRACE][MySQLWriter][PIPELINE ERROR][ColdRoomTemperatures_Archive][MySQL][MySQL]: Readable:[undefined,undefined],MySQLParser:[16,10],MySQLOutputManager:[16,16],MySQLWriter:[undefined,0]
and this output with a the 'hack' in place
[TRACE][MySQLWriter][PIPELINE ERROR][ColdRoomTemperatures_Archive][MySQL][MySQL]: Readable:[16,undefined],MySQLParser:[16,10],MySQLOutputManager:[16,16],MySQLWriter:[undefined,0]
So does imply that readable-stream is not pickup later version of steam.
from mysql.
Is the two outputs supported to be different? They both look the same to me, showing undefined
, right? Am I missing something?
from mysql.
Sorry that was cut and paste error - updated . Basically this is a snapshot of the readableLength and writableLength values for each member of a pipeline when I deliberately break the pipeline by killing the database connection used by the MySQLWriter class.
from mysql.
Ok, gotcha. In parallel I was looking at readable-stream and it looks like readableLength was added in the 3.0 release.
from mysql.
Does that version also add destroy()/_destroy. I have a project (YADAMU) that does parallel copies of data between different databases. I need the ability to terminate all active copies, if one of them fails. I do this by invoking destroy() on the reader of all the active Pipelines, it seems to work fine with most of the databases I support but doesn't seem to work with MySQL. If destroy is not implemented that would probably explain it (and indicate that I need to double check my error handling :))
from mysql.
That I'm not sure, but you're always welcome to contribute any fixes or such that you think makes sense.
from mysql.
Looks like I currently have readable-steam 2.3.7 (presumably as a result of MySQL's dependency). I just forced an install of readable-stream@latest and got 3.6.0 by the look of it. However when I revert my change to query.js and re-run the test I'm back to undefined by the look of it
[TRACE][MySQLWriter][PIPELINE ERROR][ColdRoomTemperatures_Archive][MySQL][MySQL]: Readable:[undefined,undefined],MySQLParser:[16,13],MySQLOutputManager:[16,16],MySQLWriter:[undefined,0]
Not sure if that's a valid test ?
Also looks like the package has not been updated for 17 months. I think there have been a lot of changes to the core streams implementation in that time
from mysql.
@dougwilson Doug
Not sure what to propose as the fix here. Changing the import to 'streams' seems to resolve the issue. Explicitly upgrading the 'readable-stream' component to 3.6.7 does not appear to resolve the issue. The logic behind using 'readable-stream' rather than directly using using 'streams' seems to make sense, but the package does not appear to have been upgraded for over 18 months.
If I propose switching mysql to directly use the 'readable' supplied by streams that could cause issues for consumers who are running with older versions of 'node'. Unfortunately, based on many years of experience, I suspect there are many more consumers sticking to older versions compared to consumers who track the latest one.
How diffiicult is it to change the require to use streams and run regression tests to see what if any side effects there would be to that change
Any advice much appreciated.
from mysql.
Related Issues (20)
- 腾讯云系统centos7.2安装mysql失败,怎么办麽、 HOT 1
- Error connecting to Aurora DB using the "Amazon RDS" profile HOT 4
- mysql.createPool transaction with loop HOT 1
- Error: ER_NOT_SUPPORTED_AUTH_MODE: Client does not support authentication protocol requested by server; consider upgrading MySQL client HOT 1
- How to catch ER_BAD_FIELD_ERROR HOT 2
- Missing information in Documentation HOT 1
- Multiple insert using array notation HOT 1
- Uncaught exception for "Connection lost: The server closed the connection." HOT 3
- Node 17.4, MySQL Server 8.0, Mac OS Monterey, ECONNREFUSED with error code -61, ::1:3306 HOT 5
- breaks scope HOT 3
- Supported MySQL Versions HOT 2
- Question - Get query of row HOT 2
- Write cluster failover, pool stuck with read-only cluster HOT 1
- The correct table name when creating the table, with an error: ER_WRONG_TABLE_NAME. HOT 4
- Better type casting of BIT values HOT 3
- the connection string is parsed wrongly HOT 1
- Updating URL parsing to WHATWG URL HOT 2
- RangeError: The value of "offset" is out of range. It must be >= 0 and <= 5254. Received 5261
- getConnection always get timed out
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mysql.