分页分sheet。换BigExcel

今天试玩多线程:

 List<EhrCheckworkSituationDto> listn=new CopyOnWriteArrayList<>();
            CountDownLatch latch = new CountDownLatch(dates.size());
            ExecutorService hotelExecutor = ThreadPoolConfig.hotelExecutor();
            for (String date : dates) {
                hotelExecutor.execute(
                        () -> {
                            try {
                                params.put("startDate", date);
                                params.put("endDate", date);
                                List<EhrCheckworkSituationDto> templistn = ehrCheckworkCheckinRecrodDao.getStaffList(params);
                                listn.addAll(templistn);
                            } finally {
                                latch.countDown();
                            }
                        });

            }
            try {
                latch.await();
            } catch (Exception e) {
                e.printStackTrace();
            }
            hotelExecutor.shutdown();

线程应该没关系哈,问题可能出在你们是对同一个map的引用   。有当时有问题的时候,你看到那个日期呀,都变成是同一天吗,相当于几百个线程查的都是26号那一天?。  然后map又存在key同名会替换的问题。  
改正: 就是把它放在主线程去put,而不要放在各个线程再去put,互相有这种争抢的可能性(也不能这样,也会有主与其他线程争抢的情况!所以还得是new局部变量!)

修改:

   List<EhrCheckworkSituationDto> listn=new CopyOnWriteArrayList<>();
            CountDownLatch latch = new CountDownLatch(dates.size());
            ExecutorService hotelExecutor = ThreadPoolConfig.hotelExecutor();
            for (String date : dates) {
                hotelExecutor.execute(
                        () -> {
                            try {
                                HashMap<String, Object> tempMap = new HashMap<>();
                                tempMap.putAll(params);
                                tempMap.put("startDate", date);
                                tempMap.put("endDate", date);
                                List<EhrCheckworkSituationDto> templistn = ehrCheckworkCheckinRecrodDao.getStaffList(tempMap);
                                listn.addAll(templistn);
                            } finally {
                                latch.countDown();
                            }
                        });

            }
            try {
                latch.await();
            } catch (Exception e) {
                e.printStackTrace();
            }
            hotelExecutor.shutdown();

大数据量适合多线程切割再批量插入:

  //大数据量适合mapper自定义foreach  todo 测一下哪个快
        // ehrCheckworkAttendanceDetailsService.insertBatch(notNullList);  //1953条 9793ms
        saveBatch(notNullList,1000);   // 1953条 5034ms
/**
     * 多线程+批量插入
     * @param list
     * @param dealSize
     */
    private void saveBatch(List<EhrCheckworkAttendanceDetailsEntity> list, int dealSize) {
        int totalSize = list.size();
        //线程数
        int runSize = (totalSize / dealSize) + 1;

        CountDownLatch latch = new CountDownLatch(runSize);


        for (int i = 0; i < runSize; i++) {
            //计算每个线程执行的列表数据
            int startIndex = (i * dealSize);
            List<EhrCheckworkAttendanceDetailsEntity> threadList;
            if ((i + 1) == runSize) {
                int endIndex = totalSize;
                threadList = list.subList(startIndex, endIndex);
            } else {
                int endIndex = (i + 1) * dealSize;
                threadList = list.subList(startIndex, endIndex);
            }

            executor.execute(
                    () -> {
                        try {
                            ehrCheckworkAttendanceDetailsService.insertBatch(threadList);
                        } finally {
                            latch.countDown();
                        }
                    });
        }

        try {
            latch.await();
        } catch (Exception e) {
            e.printStackTrace();
        }

    }

Logo

永洪科技,致力于打造全球领先的数据技术厂商,具备从数据应用方案咨询、BI、AIGC智能分析、数字孪生、数据资产、数据治理、数据实施的端到端大数据价值服务能力。

更多推荐