Giter Club home page Giter Club logo

zack.efcore.batch's Introduction

Zack.EFCore.Batch

中文文档 Chinese version

Using this library, Entity Framework Core users can delete or update multiple records from a LINQ Query in a SQL statement without loading entities. This libary supports Entity Framework Core 5.0 and Entity Framework Core 6.0.

Instructions:

Step 1

As for.NET 5 users:

SQLServer: Install-Package Zack.EFCore.Batch.MSSQL
MySQL: Install-Package Zack.EFCore.Batch.MySQL.Pomelo
Postgresql: Install-Package Zack.EFCore.Batch.Npgsql
Sqlite: Install-Package Zack.EFCore.Batch.Sqlite
Oracle:Install-Package Zack.EFCore.Batch.Oracle
Dm(达梦): Install-Package ZackEFCore.Batch.Dm
In Memory:Install-Package Zack.EFCore.Batch.InMemory

As For.NET 6 users:

SQLServer: Install-Package Zack.EFCore.Batch.MSSQL_NET6
MySQL: Install-Package Zack.EFCore.Batch.MySQL.Pomelo_NET6
Postgresql: Install-Package Zack.EFCore.Batch.Npgsql_NET6
Sqlite: Install-Package Zack.EFCore.Batch.Sqlite_NET6
Oracle: Install-Package Zack.EFCore.Batch.Oracle_NET6
In Memory:Install-Package Zack.EFCore.Batch.InMemory_NET6

Support of MySQL is based on Pomelo.EntityFrameworkCore.MySql.

Step 2:

Depending on the database, add the following code into OnConfiguring() method of your DbContext respectively.

optionsBuilder.UseBatchEF_MSSQL();// as for MSSQL Server
optionsBuilder.UseBatchEF_Npgsql();//as for Postgresql
optionsBuilder.UseBatchEF_MySQLPomelo();//as for MySQL
optionsBuilder.UseBatchEF_Sqlite();//as for Sqlite
optionsBuilder.UseBatchEF_Oracle();//as for Oracle
optionsBuilder.UseBatchEF_DM();//as for DM(达梦)
optionsBuilder.UseBatchEF_InMemory();//as for In Memory
Step 3:

Use the extension method DeleteRangeAsync() of DbContext to delete a set of records. The parameter of DeleteRangeAsync() is the lambda expression of the filter Example code:

await ctx.DeleteRangeAsync<Book>(b => b.Price > n || b.AuthorName == "zack yang"); 

The code above will execute the following SQL statement on database:

Delete FROM [T_Books] WHERE ([Price] > @__p_0) OR ([AuthorName] = @__s_1)

and the DeleteRange() is the synchronous version of DeleteRangeAsync().

Use the extension method BatchUpdate() of DbContext to create a BatchUpdateBuilder. There are four methods in BatchUpdateBuilder as follows

  • Set() is used for assigning a value to a property. The first parameter of the method is the lambda expression of the property, and the second one is the lambda expression of the value.
  • Where() is used for setting the filter expression
  • ExecuteAsync() is an asynchronous method that can execute the BatchUpdateBuilder, and the Execute() is a synchronous alternative of ExecuteAsync()

Example code:

await ctx.BatchUpdate<Book>()
    .Set(b => b.Price, b => b.Price + 3)
    .Set(b => b.Title, b => s)
    .Set(b => b.AuthorName,b=>b.Title.Substring(3,2)+b.AuthorName.ToUpper())
    .Set(b => b.PubTime, DateTime.Now)
    .Where(b => b.Id > n || b.AuthorName.StartsWith("Zack"))
    .ExecuteAsync();

The code above will execute the following SQL statement on database:

Update [T_Books] SET [Price] = [Price] + 3.0E0, [Title] = @__s_1, [AuthorName] = COALESCE(SUBSTRING([Title], 3 + 1, 2), N'') + COALESCE(UPPER([AuthorName]), N''), [PubTime] = GETDATE()
WHERE ([Id] > @__p_0) OR ([AuthorName] IS NOT NULL AND ([AuthorName] LIKE N'Zack%'))

Take(), Skip()

Take() and Skip() can be used to limit the affected rows of DeleteRangeAsync and BatchUpdate:

await ctx.Comments.Where(c => c.Article.Id == id).OrderBy(c => c.Message)
.Skip(3).DeleteRangeAsync<Comment>(ctx);
await ctx.Comments.Where(c => c.Article.Id == id).Skip(3).Take(10).DeleteRangeAsync<Comment>(ctx);
await ctx.Comments.Where(c => c.Article.Id == id).Take(10).DeleteRangeAsync<Comment>(ctx);

await ctx.BatchUpdate<Comment>().Set(c => c.Message, c => c.Message + "abc")
	.Where(c => c.Article.Id == id)
	.Skip(3)
	.ExecuteAsync();

await ctx.BatchUpdate<Comment>()
	.Set(c => c.Message, c => "abc")
	.Where(c => c.Article.Id == id)
	.ExecuteAsync();
	
await ctx.BatchUpdate<Comment>()
	.Set("Message","abc")
	.Where(c => c.Article.Id == id)
	.ExecuteAsync();

await ctx.BatchUpdate<Comment>().Set(c => c.Message, c => c.Message + "abc")
	.Where(c => c.Article.Id == id)
	.Skip(3)
	.Take(10)
	.ExecuteAsync();
await ctx.BatchUpdate<Comment>().Set(c => c.Message, c => c.Message + "abc")
   .Where(c => c.Article.Id == id)
   .Take(10)
   .ExecuteAsync();

BulkInsert

At this point, BulkInsert cannot be supported on SqlLite.

List<Book> books = new List<Book>();
for (int i = 0; i < 100; i++)
{
	books.Add(new Book { AuthorName = "abc" + i, Price = new Random().NextDouble(), PubTime = DateTime.Now, Title = Guid.NewGuid().ToString() });
}
using (TestDbContext ctx = new TestDbContext())
{
	ctx.BulkInsert(books);
}

On mysql, to use BulkInsert, please enable local_infile on server side and client side: enable "local_infile=ON" on mysql server, and add "AllowLoadLocalInfile=true" to connection string on client side.

Misc

This library utilizes the EF Core to translate the lambda expression to SQL statement, so it supports nearly all the lambda expressions which EF Core supports.

The following databases have been tested that they can work well with Zack.EFCore.Batch: MS SQLServer(Microsoft.EntityFrameworkCore.SqlServer), MySQL(Pomelo.EntityFrameworkCore.MySql), PostgreSQL(Npgsql.EntityFrameworkCore.PostgreSQL), Oracle(Oracle.EntityFrameworkCore). In theory, as long as a database has its EF Core 5/6 Provider , the database can be supported by this library. If you are using a database that is not currently supported, please submit an issue. I can usually complete the development within one working day.

Report of this library

zack.efcore.batch's People

Contributors

akinix avatar bpceee avatar hueifeng avatar huxingye avatar yangzhongke avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

zack.efcore.batch's Issues

更新时赋值有问题

如下:
.Set(a => a.ReadedAt, DateTime.Now)
ReadAt 为 DateTime? 类型,给它赋一个DateTime类型的值时出现错误:
Expression of type 'System.DateTime' cannot be used for return type 'System.Nullable1[System.DateTime]'`

使用.Set(a => a.ReadedAt, b=> DateTime.Now) 这样的形式则没有问题。

BulkInsertAsync无法正确插入正确的对象主键

数据库:Postgresql
我写的测试代码是想插入Int32.MinValue → Int32.MinValue + 1000*1000 的主键范围.
但我测试得到的结果是 BulkInsertAsync 会使用自增的模式.

用Efcore自身的AddRangeAsync是正常的.

.net6 DeleteRange BatchUpdate 失败

如同此问题 https://github.com/yangzhongke/Zack.EFCore.Batch/issues/46#issue-1079272716

目前可以先通过设置ExecuteAsync、DeleteRangeAsync的ignoreQueryFilters: true参数,禁用QueryFilter,这样的方式绕过。如下:

var drowsdf = await context.BatchUpdate()
.Where(x => x.PlatformOrderNo == platformOrderNo)
.Set(x => x.State, x => 10)
.ExecuteAsync(ignoreQueryFilters: true);
这样QueryFilter就会被忽略了,如果想继续使用对应的QueryFilter,可以手动把过滤条件写到Where条件中。我下周再研究从根本解决的办法。

await _sysMenuMenuDistributionRep.Context.DeleteRangeAsync<Sys_MenuDistribution>(u => u.TenantId.Equals(input.TenantId), ignoreQueryFilters: true);

组件升级到 6.0.8 你这种写法也会报错(The given key '__ef_filter__p_0' was not present in the dictionary.)
退回到6.0.6就不会

does not work with navigation properties

DbContext.DeleteRangeAsync<MyTable>(table=>
					table.navProp!= null && table.navProp.SecondNavProp.ThirdNavProp== thirdNavPropInstance);

results in an error, equalityOperator not found

when changing the last navProp to it's Id and comparing with the instance.Id, it gives an 'invalid property error'
(EDIT: navPropProp had to be navProp)

执行批量修改,空指针

环境 abpvnext .net5 + mysql
代码:

var context = await GetDbContextAsync() as OrderSyncApiDbContext;
if (context != null)
{
    var drowsdf = await context.BatchUpdate<PlatformOrderInfo>()
        .Where(x => x.PlatformOrderNo == platformOrderNo)
        .Set(x => x.State, x => 10)
        .ExecuteAsync();
}

报错:
System.NullReferenceException
Object reference not set to an instance of an object.
at Zack.EFCore.Batch.Internal.BatchUpdateBuilder1.Set[TP](Expression1 name, Expression1 value) at Dk.OrderSyncApi.PlatformOrder.PlatformOrderInfoRepository.DeleteByNo(String platformOrderNo) in D:\Git\business-center-system\services-module\order-sync-api\src\Dk.OrderSyncApi.EntityFrameworkCore\PlatformOrder\PlatformOrderInfoRepository.cs:line 99 at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo) at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue1.ProceedAsync()
at Volo.Abp.Uow.UnitOfWorkInterceptor.InterceptAsync(IAbpMethodInvocation invocation)
at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func3 proceed)
at Dk.OrderSyncApi.EntityFrameworkCore.PlatformOrder.PlatformOrderInfoRepositoryTests.ZackDeleteTest() in D:\Git\business-center-system\services-module\order-sync-api\test\Dk.OrderSyncApi.EntityFrameworkCore.Tests\EntityFrameworkCore\PlatformOrder\PlatformOrderInfoRepositoryTests.cs:line 36
at Xunit.Sdk.TestInvoker1.<>c__DisplayClass48_1.<<InvokeTestMethodAsync>b__1>d.MoveNext() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 264 --- End of stack trace from previous location --- at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func1 asyncAction) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\ExecutionTimer.cs:line 48
at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 90

批量新增在实体可空类型中会出错

测试环境.net 5 + SqlServer
定义实体 public int? id { get; set; } 可空类型
调用BulkInsertAsync 会报错:
System.NotSupportedException: DataSet does not support System.Nullable<>.
at System.Data.DataColumn..ctor(String columnName, Type dataType, String expr, MappingType type)
at Zack.EFCore.Batch.Internal.BulkInsertUtils.BuildDataTable[TEntity](DbSet1 dbSet, IEnumerable1 items)
at System.Linq.MSSQLBulkInsertExtensions.BulkInsertAsync[TEntity](DbContext dbCtx, IEnumerable`1 items, CancellationToken cancellationToken)

去掉可空类型后可以批量插入成功!!!

Batch update ignore custom type conversion (Npgsql)

I use custom converter for convert from enum to string and back (storing enum in DB as string)

public class CustomEnumConverter<TEnum> : ValueConverter<TEnum, string> where TEnum : struct, Enum
{
    public CustomEnumConverter(ConverterMappingHints mappingHints = null)
        : base(toStringExpr, toEnumExpr, mappingHints) { }

    private readonly static Expression<Func<TEnum, string>> toStringExpr = x => ToDbString(x);
    private readonly static Expression<Func<string, TEnum>> toEnumExpr = x => ToEnum(x);

    public static string ToDbString(TEnum tEnum)
    {
        var enumType = tEnum.GetType();
        var memInfo = enumType.GetMember(tEnum.ToString());

        var attr = (EnumMemberAttribute)memInfo[0]
            .GetCustomAttributes(typeof(EnumMemberAttribute), false)
            .FirstOrDefault();

        return attr.Value;
    }

    public static TEnum ToEnum(string value)
    {
        foreach (var name in Enum.GetNames<TEnum>()) {
            var attr = typeof(TEnum)
                .GetRuntimeField(name)
                .GetCustomAttribute<EnumMemberAttribute>(true);
            if (attr != null && attr.Value == value) {
                return Enum.Parse<TEnum>(name);
            }
        }

        return default;
    }
}

When I use standard EFCore functions (such as context.Sess.Add(entity)), all works fine (enums stored as string),
but batchUpdate save to DB int value of enum member, not string.

    db.BatchUpdate<Entities.Sess>()
          .Set(m => m.Status, m => Enums.SessStatus.Stopreq) // '1' in DB, must be 'stopreq'
          .Execute()

My Enum and Entity

public enum SessStatus
{
    [EnumMember(Value = "active")]
    Active,

    [EnumMember(Value = "stopreq")]
    Stopreq,

    [EnumMember(Value = "stopping")]
    Stopping,
}

public class Sess
{
    ...
    public SessStatus Status { get; set; }
    ...
}

public class AppDbContext : DbContext
{
    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder
            .Entity<Sess>()
            .Property(m => m.Status)
            .HasConversion(new Converters.CustomEnumConverter<SessStatus>());

        base.OnModelCreating(modelBuilder);
    }
}

BatchUpdateBuilder<TEntity> Set<TP> 如何定义参数作为集合传

    public virtual Task UpdateRangeAsync(
        Expression<Func<TEntity, bool>> predicate,
        params KeyValuePair<Expression<Func<TEntity>>, Expression<Func<TEntity>>>[] fieldValue
    )
    {
        if (!fieldValue.Any()) return Task.CompletedTask;
        var where = predicate.And(OtherQueryCondition);
        var obj = DbContext.BatchUpdate<TEntity>().Where(where);
        foreach (var keyValuePair in fieldValue)
        {
            obj.Set(keyValuePair.Key, keyValuePair.Value);
        }

        return obj.ExecuteAsync();
    }

如上所示:当需要更新多个字段,多个字段的修改的表达式以集合的方式传过来?

谢谢!

Mysql BulkInsert, 实体中有复杂类型字段时,插入失败

比如某个字段是List类型,该字段已映射成Json约定存储,数据库中该字段类型为varchar类型,用ef插入时,会将该字段自动序列化为字符串,能保存到数据库,但是用db.BulkInsert 扩展方法,就会报错不支持,可能是内部只处理的简单类型的字段,复杂类型的没安模型的配置处理。
我估计其他数据库也会报错吧。

使用杨老师的库之后原有的查询操作无法进行

我用的是SqlServer数据库,加了options.UseBatchEF(); 之后查询语句后面多了"FETCH FIRST 1 ROWS ONLY"
报错信息如下:

Microsoft.EntityFrameworkCore.Query[10100] An exception occurred while iterating over the results of a query for context type 'YQ.EntityFramework.AppDbContext'. Microsoft.Data.SqlClient.SqlException (0x80131904): 在 FETCH 语句中选项 FIRST 的用法无效。 at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__169_0(Task1 result)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke() at System.Threading.Tasks.Task.<>c.<.cctor>b__277_0(Object obj) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.MoveNextAsync() ClientConnectionId:e42047e6-9410-491c-8518-1dec7d2b414d Error Number:153,State:2,Class:15 Microsoft.Data.SqlClient.SqlException (0x80131904): 在 FETCH 语句中选项 FIRST 的用法无效。 at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__169_0(Task1 result)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke() at System.Threading.Tasks.Task.<>c.<.cctor>b__277_0(Object obj) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.MoveNextAsync() ClientConnectionId:e42047e6-9410-491c-8518-1dec7d2b414d

相比于EF-Plus和EFCore.BulkExtensions的更新方法来说更繁琐

当前的更新语句:

await ctx.BatchUpdate<Book>()
    .Set(b => b.Price, b => b.Price + 3)
    .Set(b => b.Title, b => s)
    .Set(b => b.AuthorName,b=>b.Title.Substring(3,2)+b.AuthorName.ToUpper())
    .Set(b => b.PubTime, b => DateTime.Now)
    .Where(b => b.Id > n || b.AuthorName.StartsWith("Zack"))
    .ExecuteAsync();

对比EF-Plus:

await ctx.Where(b => b.Id > n || b.AuthorName.StartsWith("Zack"))
    .UpdateAsync(b => new Book {
        Price = b.Price + 3,
        Title = s,
        AuthorName = b.Title.Substring(3,2)+b.AuthorName.ToUpper(),
        PubTime = DateTime.Now
    });

以及BatchExtensions:

await ctx.Where(b => b.Id > n || b.AuthorName.StartsWith("Zack"))
    .BatchUpdateAsync(b => new Book {
        Price = b.Price + 3,
        Title = s,
        AuthorName = b.Title.Substring(3,2)+b.AuthorName.ToUpper(),
        PubTime = DateTime.Now
    });

以上方法生成的都是只更新修改的字段
但是后面两个不需要每个属性都是用表达式的方式

Deleting based on Navigation Properties generates wrong SQL Statement

Hi,

I'm using EfCore with MSSQL and i'm trying to delete records belonging to a particular entity, however when I use DeleteRangeAsync it seems to skip the whole path of properties and just treats the last property as if it belonged to the root object.

The relations are like this:
User <1:N> License
License <1:N> Environment

Now when i DeleteRangeAsync:

await _dbContext.DeleteRangeAsync<Environment>(e => e.License.User.Id == id);
await _dbContext.DeleteRangeAsync<License>(l => l.User.Id == id);

it leads to the following output in SQL Server Profiler

exec sp_executesql N'Delete FROM [Environments] WHERE [Id] = @__id_0',N'@__id_0 bigint',@__id_0=3
exec sp_executesql N'Delete FROM [Licenses] WHERE [Id] = @__id_0',N'@__id_0 bigint',@__id_0=3

So it basically compares the provided id to Environment.Id and License.Id respectively, whereas the statements should actually look something like this:

DELETE env
FROM Environments AS env
INNER JOIN Licenses AS lic ON lic.Id = env.LicenseId
WHERE lic.UserId = 3;

DELETE FROM Licenses WHERE UserId = 3;

Would be great if we could find a solution for this. Thanks!

Best Regards,

Josh

E: Edited for proper syntax highlightning

引用Mysql如下未找到key值

项目为abpvnext model的项目,引用如下:
image
错误如下:
System.Collections.Generic.KeyNotFoundException
The given key '__ef_filter__p_0' was not present in the dictionary.
at System.Collections.Generic.Dictionary2.get_Item(TKey key) at Microsoft.EntityFrameworkCore.Query.SqlNullabilityProcessor.VisitSqlParameter(SqlParameterExpression sqlParameterExpression, Boolean allowOptimizedExpansion, Boolean& nullable) at Microsoft.EntityFrameworkCore.Query.SqlNullabilityProcessor.Visit(SqlExpression sqlExpression, Boolean allowOptimizedExpansion, Boolean preserveNonNullableColumns, Boolean& nullable) at Microsoft.EntityFrameworkCore.Query.SqlNullabilityProcessor.VisitSqlBinary(SqlBinaryExpression sqlBinaryExpression, Boolean allowOptimizedExpansion, Boolean& nullable) at Microsoft.EntityFrameworkCore.Query.SqlNullabilityProcessor.Visit(SqlExpression sqlExpression, Boolean allowOptimizedExpansion, Boolean preserveNonNullableColumns, Boolean& nullable) at Microsoft.EntityFrameworkCore.Query.SqlNullabilityProcessor.VisitSqlBinary(SqlBinaryExpression sqlBinaryExpression, Boolean allowOptimizedExpansion, Boolean& nullable) at Microsoft.EntityFrameworkCore.Query.SqlNullabilityProcessor.Visit(SqlExpression sqlExpression, Boolean allowOptimizedExpansion, Boolean preserveNonNullableColumns, Boolean& nullable) at Microsoft.EntityFrameworkCore.Query.SqlNullabilityProcessor.Visit(SqlExpression sqlExpression, Boolean allowOptimizedExpansion, Boolean& nullable) at Microsoft.EntityFrameworkCore.Query.SqlNullabilityProcessor.Visit(SelectExpression selectExpression) at Microsoft.EntityFrameworkCore.Query.SqlNullabilityProcessor.Process(SelectExpression selectExpression, IReadOnlyDictionary2 parameterValues, Boolean& canCache)
at Pomelo.EntityFrameworkCore.MySql.Query.Internal.MySqlParameterBasedSqlProcessor.ProcessSqlNullability(SelectExpression selectExpression, IReadOnlyDictionary2 parametersValues, Boolean& canCache) at Microsoft.EntityFrameworkCore.Query.RelationalParameterBasedSqlProcessor.Optimize(SelectExpression selectExpression, IReadOnlyDictionary2 parametersValues, Boolean& canCache)
at System.Linq.BatchEFExtensions.Parse[TEntity](IQueryable1 queryable, DbContext ctx, Boolean ignoreQueryFilters) at System.Linq.BatchEFExtensions.GenerateDeleteSQL[TEntity](DbContext ctx, IQueryable1 queryable, Expression1 predicate, Boolean ignoreQueryFilters, IReadOnlyDictionary2& parameters)
at System.Linq.BatchEFExtensions.DeleteRangeAsync[TEntity](DbContext ctx, Expression1 predicate, Boolean ignoreQueryFilters, CancellationToken cancellationToken) at Dk.OrderSyncApi.PlatformOrder.PlatformOrderInfoRepository.DeleteByNo(String platformOrderNo) in D:\Git\business-center-system\services-module\order-sync-api\src\Dk.OrderSyncApi.EntityFrameworkCore\PlatformOrder\PlatformOrderInfoRepository.cs:line 96 at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo) at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue1.ProceedAsync()
at Volo.Abp.Uow.UnitOfWorkInterceptor.InterceptAsync(IAbpMethodInvocation invocation)
at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func3 proceed)
at Dk.OrderSyncApi.EntityFrameworkCore.PlatformOrder.PlatformOrderInfoRepositoryTests.ZackDeleteTest() in D:\Git\business-center-system\services-module\order-sync-api\test\Dk.OrderSyncApi.EntityFrameworkCore.Tests\EntityFrameworkCore\PlatformOrder\PlatformOrderInfoRepositoryTests.cs:line 36
at Xunit.Sdk.TestInvoker1.<>c__DisplayClass48_1.<<InvokeTestMethodAsync>b__1>d.MoveNext() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 264 --- End of stack trace from previous location --- at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func1 asyncAction) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\ExecutionTimer.cs:line 48
at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 90

.net6 DeleteRange删除失败

图片
zack.efcore.batch.npgsql_net6版本6.0.5
Npgsql数据库,Device中是设置了主键的
opt.UseBatchEF_Npgsql(); // EF批量组件

Is there a way to delete with a limit?

For example with other libraries in the past I've used:

            await _context.MyTable
                .Where(x => x.Started < expireDate)
                .Take(50)
                .DeleteAsync();

Is something like this possible?

Table should be specified

Current syntax:

dbContext.DeleteRangeAsync<Book>(...)

The issue will appear if we have two tables with a class Book. It won't be clear from which table the items will be deleted (if any). There is a better approach to specify the collection from which the entries should be removed.

dbContext.Books.DeleteRangeAsync(...)

BulkInsert 返回 311 rows were copied to xxx but only 1 were inserted.

老师您好,
BulkInsert 方法一直报错,关于这个错误我有把您的源码下载下来跑,但是错误好像是 MySqlConnector 里抛出的。
这个项目里的Issues也有类似的情况,需要修改链接字符串UseAffectedRows=true 但是还是报错。。。
实在是没办法了,最后还是只能在老师您这提个Issues

Support for NpgSql Schema

Please add support for the schemas . Right now the library assumes that all tables reside in the default database schema and that may not be the case.

Example of an EF Entity with a custom Schema:

[Table("persons", Schema = "catalog")]
public class Person {
    [Key]
    public int Id { get; set; }
}

In order to delete from this table the query would need to be:

DELETE FROM catalog.persons WHERE ....

Right now it's trying to do:

DELETE FROM persons WHERE ....

This causes an error saying the table 'persons' does not exist.

I'm testing with the NpgSql version.

mysql版本中,使用BulkInsertAsync会丢失Datetime的时区信息,变成未知时区

如题,mysqlconnector那边对于时区与连接串时区不匹配时,会抛出异常。我修改了mysqlconnector的代码,使其强制按连接串时区处理,但发现Zack.EFCore.Batch.MySQL.Pomelo在调用MySqlBulkCopy时,已经丢失了时区信息,变为了DateTimeKind.Unspecified,我不得不在批量提交前反射时间类型的数据,强制UTC

protected void toUTCDateTime(T entity)
        {
            //反射每一个时间类型的字段,修改其时区
            Type t = typeof(T);
            foreach (var p in t.GetProperties())
            {
                if ((p.PropertyType.IsGenericType && p.PropertyType.GenericTypeArguments[0].Name == "DateTime") || p.PropertyType.Name == "DateTime")
                {
                    object o = p.GetValue(entity);
                    if (o != null)
                    {
                        DateTime d = (DateTime)o;
                        if (d.Kind != DateTimeKind.Utc)
                            p.SetValue(entity, d.ToUniversalTime());
                    }
                }
            }
            
        }

是否支持联表更新或删除。

环境 .NET 5 + SQL Server

比如sql 中的 联表更新
UPDATE TableA SET Column1= '测试'
FROM TableB AS A
INNER JOIN TableA AS B ON A.ID = B.ID
WHERE A.Type ='A' AND B.SType = 'B'

尝试在linq中写
ctx.BatchUpdate<表1>()
.Set(a => a.Column1, a => "测试")
.Where(a => a.Type == "A")
.Where(a => ctx.TableB.Where(b => b.ID == a.ID && b.SType == "B").Any())
.ExecuteAsync();
会报错 The count of columns should be even.

希望能够支持第三方的Oracle Provider:Devart.Data.Oracle.EFCore

  目前Zack.EFCore.Batch支持的是Oracle的官方Provider,但是官方的Provider没有实现NetTopologySuite,难以操作空间/地理数据,希望能提供对Devart.Data.Oracle.EFCore的支持。

  使用前需要先安装dotConnect for Oracle,Devart为收费软件,不过可以试用或安装Express版,下载地址:https://www.devart.com/dotconnect/oracle/download.html

  参考文档:https://www.devart.com/entitydeveloper/docs/index.html?_ga=2.123315787.1509785132.1623132137-1510001402.1616997830

批量更新时,某些字段未更新

var now = DateTime.Now;
var _48 = now.AddHours(-48);
var effect = await _dbContext.BatchUpdate<T_Evaluation>()
.Set(a => a.Attitude, b => 5)
.Set(a => a.Speed, b => 5)
.Set(a => a.Specialized, b => 5)
.Set(a => a.Note, b => "默认好评!")
.Set(a => a.ModificationTime, b => now)
.Where(a => !a.Attitude.HasValue && a.CreationTime.Value < _48)
.ExecuteAsync();

以上Attitude、Speed、Specialized三个字段均为记录打分的数字类型(int)
执行之后只有Attitude以及后面的Note、ModificationTime被更新
Speed和Specialized未更新,调换三个字段的顺序发现每次都只有第一个数字被更新。

用的包是:Zack.EFCore.Batch.MSSQL

批量更新操作的参数值为null时出现异常

比如下面的代码:
string eva = null;
await DbContext.BatchUpdate<YQ_Proj_Info>()
.Set(a => a.Evaluate, b => eva)
.Where(a => a.Id == projId)
.ExecuteAsync();

当要更新的字符串为null时,出现以下错误:
参数化查询 '(@__projId_0 nvarchar(9),@__eva_1 nvarchar(4000))Update [YQ_Proj' 需要参数 '@__eva_1',但未提供该参数。

不支持AddDbContextPool吗

我用的是Zack.EFCore.Batch.MySQL.Pomelo包,在DbContext类的OnConfiguring方法中加上optionsBuilder.UseBatchEF_MySQLPomelo()后,
查询时报异常OnConfiguring' cannot be used to modify DbContextOptions when DbContext pooling is enabled.
请问怎么解决呢

MSSQL 异常

Unhandled exception. System.MissingMethodException: Method not found: 'Microsoft.EntityFrameworkCore.Metadata.IEntityType Microsoft.EntityFrameworkCore.ModelExtensions.FindEntityType(Microsoft.EntityFrameworkCore.Metadata.IModel, System.Type)'.
at Zack.EFCore.Batch.Internal.BatchUpdateBuilder1.GenerateSQL(Expression1 predicate, Boolean ignoreQueryFilters, IReadOnlyDictionary2& parameters) at Zack.EFCore.Batch.Internal.BatchUpdateBuilder1.ExecuteAsync(Boolean ignoreQueryFilters, CancellationToken cancellationToken)
at LXD.Demo.Program.Main(String[] args) in D:\MeCode\LXD.Demo\LXD.Demo\Program.cs:line 13
at LXD.Demo.Program.

(String[] args)

Using library breaks special functions in ngpsql

npgsql/efcore.pg#1609

Functions like EF.Functions.ContainsOrEqual(arg1, arg2) incorrect translation to SQL with library active

Code for testing:

using Microsoft.EntityFrameworkCore;
using System.ComponentModel.DataAnnotations;
using System.ComponentModel.DataAnnotations.Schema;
using System.Net;
using System.Linq;

namespace XRadTest
{
    class Program
    {
        public static void Main()
        {
            var db = new AppDbContext();
            var ip = IPAddress.Parse("1.0.0.1");

            // This throw Exception
            db.User
                .Where(m => EF.Functions.ContainsOrEqual(m.IPv4.Value, ip))
                .FirstOrDefault();
        }
    }

    public class AppDbContext : DbContext
    {
        protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            optionsBuilder.UseBatchEF();
            optionsBuilder.UseNpgsql("Host=127.0.0.1;Database=test;Username=postgres;Password=;Keepalive=30");

            base.OnConfiguring(optionsBuilder);
        }

        public DbSet<User> User { get; set; }
    }

    [Table("user")]
    public class User
    {
        [Key, Column("uid")]
        public string Uid { get; set; }

        [Column("ip")]
        public (IPAddress, int)? IPv4 { get; set; }
    }
}
create table "user" (
uid varchar(128),
ip cidr,
)

Async operations must take CancellationToken as parameter

Async operations (e.g. ExecuteAsync) must take CancellationToken as parameter. Usually it is the last optional parameter.
So, you should change, for example, change

Task<int> ExecuteAsync(bool ignoreQueryFilters = false) 

to

Task<int> ExecuteAsync(bool ignoreQueryFilters = false, CancellationToken cancellationToken = default)

postgresql中bulkinsert中的事务问题

你好!
我在使用批量插入的时候, 发现bulkinsert方法中开启了新的事务.
var conn = dbCtx.Database.GetDbConnection(); conn.OpenIfNeeded(); DbTransaction tx = conn.BeginTransaction(); using (tx) { try { using (var writer = BuildImporter<TEntity>(dbCtx, (NpgsqlConnection)conn, items)) { writer.Complete(); writer.Close(); } tx.Commit(); } catch { tx.Rollback(); throw; } }
因为我使用了多个插入方式, 想保证事务的完整性. 不知道这个问题是属于bug, 还是使用上方式不对.
另外我查看了其他的数据库的bulkinsert, 发现并没有重新begintrans.

Skip and Take don't seem to work

In your docs, you show a sample using Skip and Take in a bulk delete operation:

await ctx.Comments.Where(c => c.Article.Id == id).Skip(3).Take(10).DeleteRangeAsync<Comment>(ctx);

However, when executing this locally, I don't see any trace of the Skip and Take in the generated SQL:

Delete FROM [Blogs] WHERE [Id] > 3

What SQL where you thinking of generating for translating Skip/Take for DELETE? Also, since there's no OrderBy in your sample, the Skip/Take operators are non-deterministic.

Code sample
await using var ctx = new BlogContext();
await ctx.Database.EnsureDeletedAsync();
await ctx.Database.EnsureCreatedAsync();

_ = await ctx.Blogs.Where(b => b.Id > 3).Skip(2).Take(3).DeleteRangeAsync(ctx);

public class BlogContext : DbContext
{
    public DbSet<Blog> Blogs { get; set; }

    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        => optionsBuilder
            .UseSqlServer(@"Server=localhost;Database=test;User=SA;Password=Abcd5678;Connect Timeout=60;ConnectRetryCount=0")
            .UseBatchEF_MSSQL()
            .LogTo(Console.WriteLine, LogLevel.Information)
            .EnableSensitiveDataLogging();

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
    }
}

public class Blog
{
    public int Id { get; set; }
    public string Name { get; set; }
}

解决Value Converter的支持异常

我下载最新的 Zack.EFCore.Batch.MySql.Pomelo(1.5.1)
在断点还没有走到BulkInsert时,之前的批量删除方法报错了,而且我用ef的Add也会报相同的错误 ,之前除了批量数据落地功能外,修改跟删除是可以用的,而且也不会影响现有ef的增删改,本次更新的包按要求都升到了net 5.0
DAC9A2D7-B916-427B-A745-63A21246B0E9
Uploading BB3F8C34-D8E5-4A5A-A867-04D9AECF6BC9.jpeg…

Is there a way to setup batching?

For example with other libraries in the past I've used:

            var batchSize = 1000;
            await _context.MyTable
                .Where(x => x.Started < expireDate)
                .DeleteAsync(x => x.BatchSize = batchSize);

Is something like this possible? (So it will perform a delete in batches of 1000 at once, to ensure there aren't too many being deleted in a single call, causing a timeout)

BulkInsert options.InsertIfNotExists

It would be nice to add support for inserting objects only if they do not exist.
For example, entityframework-extensions it looks like this:
context.BulkInsert(customers, options => { options.InsertIfNotExists = true; });

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.