Java uses tool classes to improve the efficiency of writing reports

The tool class and demo code repository in this article

Why use java code to write reports

For report data, in most cases, the method of writing sql is used to provide data sources for large screens/reports, but in some complex cases, it cannot be achieved by using only sql, or when it is difficult to achieve, it will be adopted to implement complex logic through code. Return the result.

problems encountered

For relatively complex reports, it is often necessary to do data connection, that is, table-to-table join, grouping, calculation and other operations. sql naturally supports these operations, and it is very easy to implement. But when we need to connect data in java code, the native support is not so friendly, we often do this

There are now two sets

List<ContractDetail> contractDetails; // A collection of contract details, the contract will be repeated
List<ContractInfo> contractInfos; // The main information of the contract, there will be no duplicate contracts

Corresponding data structure

public class ContractDetail {
      /**
     * Contract Number
     */
    private String contractNo;

        /**
     * lump sum
     */
    private BigDecimal moneyTotal;
}

public class ContractInfo {
      /**
     * Contract Number
     */
    private String contractNo;

        /**
     * state
     */
    private String status;
}





need
contractDetails is associated with contractInfos according to contractNo, filtering out the data with status = 'signed'
Then group by contractNo in contractDetails, and find the sum of moneyTotal corresponding to each contractNo
The final output should be a map

  Map<String /* contract code */, BigDecimal /* Corresponding to the sum of moneyTotal */> result;

Usually we do this

//  setp 1 filter out the contract code in the signed state
Set<String> stopContract = contractInfos.stream()
                .filter(it -> "signed".equals(it.getStatus()))
                .map(ContractInfo::getContractNo).collect(Collectors.toSet());


//step2 According to the contract code set of step1, filter out the contractDetail with the correct status
  contractDetails = contractDetails.stream()
                .filter(it -> stopContract.contains(it.getContractNo()))
                .collect(Collectors.toList());

//step3 Accumulate the corresponding moneyTotal according to contractNo
 Map<String, BigDecimal> result = new HashMap<>();
 contractDetails.stream().forEach(it -> {
            BigDecimal moneyTotal = Optional.ofNullable(result.get(it.getContractNo()))
                    .orElse(BigDecimal.ZERO);
            moneyTotal = moneyTotal.add(it.getMoneyTotal() != null ? it.getMoneyTotal() : BigDecimal.ZERO);
            result.put(it.getContractNo(), moneyTotal);
        });

Obviously, this implementation is more complicated, because using sql is nothing more than join and group by grouping after the connection. beg for peace. This problem can be easily solved. Then look at the latter tool class, and then think about whether there is an easier way to achieve it.

Tools

CollectionDataStream

The function of the collection data stream CollectionDataStream is to associate the collections through the interface, and realize two operations similar to sql join and left join
And realize the function of mutual conversion with Stream in java.
The aggregate data structure converts the collection into a data structure similar to the table structure, including the table name, data

public class AggregationData {
    Map<String, Map> aggregationMap;

    private AggregationData(){
        aggregationMap = new HashMap<>();
    }

    //key is an alias, value is the corresponding object
    public AggregationData(String tableName, Object data) {
        aggregationMap = new HashMap<>();
        aggregationMap.put(tableName, BeanUtil.beanToMap(data));
    }

    public Map<String, Map> getRowAllData() {
        return aggregationMap;
    }

    public Map getTableData(String tableName) {
        if (!aggregationMap.containsKey(tableName)) {
            throw new DataStreamException(tableName + ".not.exists");
        }
        return aggregationMap.get(tableName);
    }

    public void setTableData(String tableName, Object data) {
        if(aggregationMap.containsKey(tableName)){
            throw new DataStreamException(tableName+".has.been.exists!");
        }
        aggregationMap.put(tableName, BeanUtil.beanToMap(data));
    }


    private void setTableData(String tableName, Map<String, Object> data) {
        Map<String, Object> tableData =
                Optional.ofNullable(aggregationMap.get(tableName)).orElse(new HashMap<String, Object>());
        tableData.putAll(data);
        aggregationMap.put(tableName, tableData);
    }

    public AggregationData copyAggregationData() {
        AggregationData aggregationData = new AggregationData();
        for (String tableName : this.getRowAllData().keySet()) {
            aggregationData.setTableData(tableName, this.getRowAllData().get(tableName));
        }
        return aggregationData;
    }
}

AggregationData represents a row of data, the key of aggregationMap is the table name, and the value is the corresponding data
Let's take a closer look at this interface

import java.util.Collection;
import java.util.Map;
import java.util.function.Function;
import java.util.stream.Stream;

public interface CollectionDataStream<T> {

    /**
     *Convert the collection to a data stream and give it an alias
     * @param tableName
     * @param collection
     * @return
     */
    static CollectionDataStream<AggregationData> of(String tableName, Collection<?> collection) {
        return new CollectionDataStreamImpl(tableName, collection);
    }
    /**
     *Convert Stream to data stream and give an alias
     * @param tableName
     * @param collection
     * @return
     */
    static CollectionDataStream<AggregationData> of(String tableName, Stream<?> collection) {
        return new CollectionDataStreamImpl(tableName, collection);
    }

    /**
     * Inner join, customizable join condition, using double loop
     *
     * @param tableName
     * @param collection
     * @param predict
     * @param <T1>
     * @return
     */
    <T1> CollectionDataStream<T> join(String tableName, Collection<T1> collection, JoinPredicate<T, T1> predict);

    /**
     * Equivalent inner join, using map optimization
     *
     * @param collection
     * @param tableName
     * @param aggregationMapper
     * @param dataValueMapper
     * @param <T1>
     * @param <R>
     * @return
     */
    //Equivalent conditions recommended usage
    <T1, R> CollectionDataStream<T> joinUseHashOnEqualCondition(String tableName, Collection<T1> collection, Function<T, R> aggregationMapper, Function<T1, R> dataValueMapper);


    /**
     * Left join, customizable join condition, using double loop
     *
     * @param tableName
     * @param collection
     * @param predict
     * @param <T1>
     * @return
     */
    <T1> CollectionDataStream<T> leftJoin(String tableName, Collection<T1> collection, JoinPredicate<T, T1> predict);

    /**
     * Equivalent left join, using map optimization
     *
     * @param collection
     * @param tableName
     * @param aggregationMapper
     * @param dataValueMapper
     * @param <T1>
     * @param <R>
     * @return
     */
    <T1, R> CollectionDataStream<T> leftJoinUseHashOnEqualCondition( String tableName, Collection<T1> collection,Function<T, R> aggregationMapper, Function<T1, R> dataValueMapper);

    Stream<T> toStream();

    Stream<Map> toStream(String tableName);

    <R> Stream<R> toStream(String tableName, Class<R> clzz);


    <R> Stream<R> toStream(Function<AggregationData, R> mapper);
}

Note the difference between the joinUseHashOnEqualCondition and join methods.
If the connection between collections is an equal-value connection of a field, then use joinUseHashOnEqualCondition, which internally uses map grouping to connect. If you use join directly, the connection conditions can be customized, but the conditions are judged through double loops, which is less efficient. Therefore, in the case of equal value, it is more efficient to use joinUseHashOnEqualCondition.

how to use

Or the above requirements as an example
First perform the connection between the two sets

 CollectionDataStream.of("t1", contractDetails) .joinUseHashOnEqualCondition(
                        contractInfos.stream().filter(it -> "signed".equals(it.getStatus())).collect(Collectors.toList()),
                        "t2",
                        agg -> agg.getTableData("t1").get("contractNo"),
                        ContractInfo::getContractNo
                );

Code parsing

 CollectionDataStream.of("t1", contractDetails)

is to convert the collection contractDetails into a data stream named t1,

 .joinUseHashOnEqualCondition(
                        contractInfos.stream().filter(
                          "t2",
                            it -> "signed".equals(it.getStatus())).collect(Collectors.toList()),
                        agg -> agg.getTableData("t1").get("contractNo"),
                        ContractInfo::getContractNo
                );

Internally connect contractInfos, and alias contractInfos as t2. The connection condition is that the contractNo of t1 and the contractNol of contractInfos are connected to obtain a new aggregated data stream.

Of course, you can also use a custom connection to achieve

CollectionDataStream.of("t1", contractDetails)
                .join("t2",
                        contractInfos.stream().filter(it -> "signed".equals(it.getStatus())).collect(Collectors.toList()),
                        (agg, data) -> agg.getTableData("t1").get("contractNo").equals(data.getContractNo())
                )

Here, through the inner join, it also plays a filtering role. After the connection is completed, we also need to group for calculation, then we need to use the next tool class

MyCollectors

It is an extension of the native Collectors in stram, which implements more operations for common grouping of reports.

MyCollectorspackage collector;

import utils.NumberUtil;

import java.math.BigDecimal;
import java.util.Comparator;
import java.util.Map;
import java.util.function.Function;
import java.util.stream.Collector;
import java.util.stream.Collectors;

public class MyCollectors {
    /**
     * Returns a Collector for grouping the collection and, for multiple elements in the group, only the last one is returned, the others are ignored
     * It is suitable for the case where the key of the group is unique, and the value can be null
     * Use with caution, if there are multiple groups, data will be lost! ! !
     * @param keyMapper
     * @param <T>
     * @param <K>
     * @param <U>
     * @param <M>
     * @return
     */
    public static <T, K, U, M extends Map<K, U>>
    Collector<T, ?, Map<K, U>> groupingByLast(Function<? super T, ? extends K> keyMapper,
                                               Function<? super T, ? extends U> valueMapper) {
        return Collectors.groupingBy(keyMapper, Collectors.reducing(null, valueMapper, (o1, o2) -> o2));
    }


    /**
     * Pass in a keyMaper and a comparator
     * Group according to the key, use the comparator to compare within the group, and finally get a maximum result
     * @param keyMapper
     * @param comparator
     * @param <T>
     * @param <K>
     * @param <U>
     * @param <M>
     * @return
     */
    public static <T, K, U, M extends Map<K, U>>
    Collector<T, ?, Map<K, T>> groupingByMaxComparator(Function<? super T, ? extends K> keyMapper,
                                                      Comparator<T> comparator) {
        return Collectors.groupingBy(keyMapper, Collectors.collectingAndThen(Collectors.maxBy(comparator), it -> it.orElse(null)));
    }

    /**
     * Pass in a keyMaper and a comparator
     * Group according to the key, use the comparator to compare within the group, and finally get a minimum result
     * @param keyMapper
     * @param comparator
     * @param <T>
     * @param <K>
     * @param <U>
     * @param <M>
     * @return
     */
    public static <T, K, U, M extends Map<K, U>>
    Collector<T, ?, Map<K, T>> groupingByMinComparator(Function<? super T, ? extends K> keyMapper,
                                                       Comparator<T> comparator) {
        return Collectors.groupingBy(keyMapper, Collectors.collectingAndThen(Collectors.maxBy(comparator), it -> it.orElse(null)));
    }


    /**
     * After grouping, the group is summed according to the specified field
     * @param keyMapper
     * @param <T>
     * @param <K>
     * @return
     */
    public static <T, K>
    Collector<T, ?, Map<K, BigDecimal>> groupingAndSum(Function<? super T, ? extends K> keyMapper,
                                                       Function<? super T, BigDecimal> valueMapper) {
        return Collectors.groupingBy(keyMapper, Collectors.reducing(BigDecimal.ZERO, valueMapper, NumberUtil::addNumbers));
    }



    /**
     * Summation based on a field of an object
     * @param mapper
     * @param <T>
     * @return
     */
    public static <T>
    Collector<T, ?, BigDecimal> sumByField(Function<? super T, ? extends BigDecimal> mapper) {
        return Collectors.reducing(BigDecimal.ZERO, mapper, NumberUtil::addNumbers);
    }

    /**
     * sum
     */
    public static Collector<BigDecimal, ?, BigDecimal> sum() {
        return Collectors.reducing(BigDecimal.ZERO, NumberUtil::addNumbers);
    }
}



Implementations used in combination

 Map<String /* Has the contract changed? */, BigDecimal /* Corresponding to the sum of moneyTotal */> result = CollectionDataStream.of("t1", contractDetails)
                .joinUseHashOnEqualCondition(
                        contractInfos.stream().filter(it -> "60".equals(it.getStatus())).collect(Collectors.toList()),
                        "t2",
                        agg -> agg.getTableData("t1").get("contractNo"),
                        ContractInfo::getContractNo
                ).toStream("s1", ContractDetail.class)//Convert data stream to java native Stream
                .collect(MyCollectors.groupingAndSum(ContractDetail::getContractNo, ContractDetail::getMoneyTotal));

This implementation is obviously simpler, reduces the probability of errors, reduces the amount of code, and improves efficiency.

Advantage

  1. The connection operation between collections is implemented, and it is a streaming operation, which can continuously connect multiple collections in one go.
  2. Realize the mutual conversion with Stream. Various complex operations, such as filtering, transformation, grouping, etc., can be realized by utilizing the functions of stream.
  3. There is a certain guarantee in terms of efficiency. Map optimization is used for equi-join, and when using inner join, consider optimizing the use of small tables and connecting large tables. In some cases, the number of cycles is reduced, and BeanMap under cglib is used when bean is copied.

If interested, the code repository address is https://github.com/404008945/dataStream

Tags: Java Database IDE programming language

Posted by patrickcurl on Sun, 23 Oct 2022 01:29:01 +0300