Excel大数据量的优化
分页分sheet。换BigExcel今天试玩多线程:List<EhrCheckworkSituationDto> listn=new CopyOnWriteArrayList<>();CountDownLatch latch = new CountDownLatch(dates.size());ExecutorService hotelExecutor = ThreadPo
·
分页分sheet。换BigExcel
今天试玩多线程:
List<EhrCheckworkSituationDto> listn=new CopyOnWriteArrayList<>();
CountDownLatch latch = new CountDownLatch(dates.size());
ExecutorService hotelExecutor = ThreadPoolConfig.hotelExecutor();
for (String date : dates) {
hotelExecutor.execute(
() -> {
try {
params.put("startDate", date);
params.put("endDate", date);
List<EhrCheckworkSituationDto> templistn = ehrCheckworkCheckinRecrodDao.getStaffList(params);
listn.addAll(templistn);
} finally {
latch.countDown();
}
});
}
try {
latch.await();
} catch (Exception e) {
e.printStackTrace();
}
hotelExecutor.shutdown();
线程应该没关系哈,问题可能出在你们是对同一个map的引用 。有当时有问题的时候,你看到那个日期呀,都变成是同一天吗,相当于几百个线程查的都是26号那一天?。 然后map又存在key同名会替换的问题。
改正: 就是把它放在主线程去put,而不要放在各个线程再去put,互相有这种争抢的可能性(也不能这样,也会有主与其他线程争抢的情况!所以还得是new局部变量!)
修改:
List<EhrCheckworkSituationDto> listn=new CopyOnWriteArrayList<>();
CountDownLatch latch = new CountDownLatch(dates.size());
ExecutorService hotelExecutor = ThreadPoolConfig.hotelExecutor();
for (String date : dates) {
hotelExecutor.execute(
() -> {
try {
HashMap<String, Object> tempMap = new HashMap<>();
tempMap.putAll(params);
tempMap.put("startDate", date);
tempMap.put("endDate", date);
List<EhrCheckworkSituationDto> templistn = ehrCheckworkCheckinRecrodDao.getStaffList(tempMap);
listn.addAll(templistn);
} finally {
latch.countDown();
}
});
}
try {
latch.await();
} catch (Exception e) {
e.printStackTrace();
}
hotelExecutor.shutdown();
大数据量适合多线程切割再批量插入:
//大数据量适合mapper自定义foreach todo 测一下哪个快
// ehrCheckworkAttendanceDetailsService.insertBatch(notNullList); //1953条 9793ms
saveBatch(notNullList,1000); // 1953条 5034ms
/**
* 多线程+批量插入
* @param list
* @param dealSize
*/
private void saveBatch(List<EhrCheckworkAttendanceDetailsEntity> list, int dealSize) {
int totalSize = list.size();
//线程数
int runSize = (totalSize / dealSize) + 1;
CountDownLatch latch = new CountDownLatch(runSize);
for (int i = 0; i < runSize; i++) {
//计算每个线程执行的列表数据
int startIndex = (i * dealSize);
List<EhrCheckworkAttendanceDetailsEntity> threadList;
if ((i + 1) == runSize) {
int endIndex = totalSize;
threadList = list.subList(startIndex, endIndex);
} else {
int endIndex = (i + 1) * dealSize;
threadList = list.subList(startIndex, endIndex);
}
executor.execute(
() -> {
try {
ehrCheckworkAttendanceDetailsService.insertBatch(threadList);
} finally {
latch.countDown();
}
});
}
try {
latch.await();
} catch (Exception e) {
e.printStackTrace();
}
}
更多推荐


所有评论(0)